XGBoost and LightGBM: The Champions of Tabular Data

You open a lending app, key in your PAN number, monthly salary, employer name, and how many EMIs you are currently paying off, then tap "Check Eligibility." Within seconds: "Congratulations! You are pre-approved for Rs 2,40,000." No loan officer read your form. No human looked at your salary slip. A model scored you in an instant, and it very likely did so using one of exactly two open-source libraries: XGBoost or LightGBM.

That might seem like an odd place to expect the cutting edge of machine learning. Neural networks get the headlines — they write essays, recognize faces, generate images. But none of that is what a lending app, a hospital's readmission-risk system, or an insurer's claim checker actually works with. What they have is a spreadsheet: one row per applicant, one column per fact about them — income, age, credit score, city, number of existing loans. Rows and columns. A tabular dataset. And on tabular data specifically, deep neural networks are rarely the first tool practitioners reach for. The original research paper introducing XGBoost reported that among 29 challenge-winning solutions published on Kaggle's own blog during 2015, 17 of them used XGBoost — more than any other single method, ahead of deep neural networks, which appeared in 11. Knowing exactly what is inside these two systems, and why it took two separate, multi-year rounds of engineering to build them, is what separates treating them as black boxes from actually knowing which setting to change when a model underperforms.

A Fast Recap: From One Tree to a Sequence of Corrections

A single decision tree is easy to picture and easy to overfit. Random Forest deals with that by growing many trees independently, each on a random slice of the data and features, and averaging their votes — independent opinions, combined. Gradient boosting takes a different route: it grows trees one at a time, in sequence, and every new tree has exactly one job — predict the errors that the current group of trees is still making, and add a small, shrunk version of that correction to the running prediction. Statistician Jerome Friedman formalized this idea in a 2001 paper in the Annals of Statistics, framing it as gradient descent carried out over functions rather than numbers: each tree is literally one step downhill.

"Gradient boosting" describes a strategy, not a finished piece of software. It does not say exactly how to pick where each tree should split, how large a correction to add, or how to stop the sequence before it starts memorizing individual training rows. XGBoost and LightGBM are two different, heavily engineered answers to exactly those questions, released two years apart, and both still training real production models today.

XGBoost: Precision Engineering for Gradient Boosting

XGBoost began as the side project of a PhD student. Tianqi Chen, working under advisor Carlos Guestrin at the University of Washington, first released XGBoost as open-source software in March 2014. It built its reputation the way a lot of ML tools do — by winning competitions — well before Chen and Guestrin wrote up the design as a formal paper, "XGBoost: A Scalable Tree Boosting System," presented at the 2016 ACM SIGKDD conference. That paper also reported that at the 2015 KDD Cup, a major industry data-mining competition, XGBoost was used by every single team that finished in the top ten.

What XGBoost added on top of plain gradient boosting comes down to two ideas.

The first is precision about how large each correction should be. Ordinary gradient boosting mostly looks at the gradient — how wrong, and in which direction, the current prediction is. XGBoost also uses the second derivative of the loss function, the Hessian, which captures how confident that gradient signal is. Using both together is a form of Newton's method from calculus, and it lets XGBoost compute the mathematically optimal size for each leaf's correction directly, instead of guessing. For the ordinary squared-error loss used in regression, the Hessian happens to work out to a constant 1 for every data point, so in practice it behaves like a simple count; the real payoff shows up once you move to the more elaborate loss functions used for classification and ranking.

The second is regularization built directly into the objective the trees are trained to minimize, rather than applied afterward as a separate pruning step. Every candidate split is scored by how much it would actually reduce the loss, after subtracting a penalty for making the tree more complex. Concretely, XGBoost scores a split using its structure-score formula:

Gain = 1/2 * [ GL^2/(HL + lambda) + GR^2/(HR + lambda) - (GL + GR)^2/(HL + HR + lambda) ] - gamma

Here GL and GR are the sums of the gradients of the data points that would land in the left and right child, HL and HR are the sums of their Hessians, lambda is an L2 penalty on how large a leaf's correction is allowed to be, and gamma is a flat cost charged for every additional leaf the tree grows. If no split can produce a positive Gain after that penalty, XGBoost simply does not make the split — regularization is what keeps individual trees shallow and cautious instead of chasing every quirk in the training data. XGBoost also handles missing values without any manual imputation: for every split, it tests sending missing values left and sending them right, and keeps whichever direction produces the higher Gain, a technique its paper calls sparsity-aware split finding. A large share of that paper is also devoted to systems engineering — storing data in compressed column blocks so the search for the best split across features can be spread across CPU cores, cache-aware memory access patterns, and the ability to train on data too large to fit in RAM. It is this combination, a sharper mathematical objective plus serious engineering, that explains why XGBoost outran the gradient boosting implementations that came before it.

Worked Example: Tracing One XGBoost Split by Hand

Suppose an NBFC wants to see how its risk model would treat five past loan applicants, using one feature to start: monthly income, in thousands of rupees. Each applicant also has a known outcome, recorded afterward as a risk score from 0 (fully safe) to 100 (defaulted badly):

• Applicant A — income Rs 20k — actual risk score 80
• Applicant B — income Rs 35k — actual risk score 65
• Applicant C — income Rs 45k — actual risk score 40
• Applicant D — income Rs 60k — actual risk score 30
• Applicant E — income Rs 90k — actual risk score 10

Step 1 — Start from the mean. Before building any tree, XGBoost's very first prediction for everyone is simply the average of the targets: (80 + 65 + 40 + 30 + 10) / 5 = 45.

Step 2 — Compute gradients. Using squared-error loss, the gradient for each point is (prediction - actual), and the Hessian is 1 for every point:

• A: 45 - 80 = -35
• B: 45 - 65 = -20
• C: 45 - 40 = 5
• D: 45 - 30 = 15
• E: 45 - 10 = 35

These five gradients sum to exactly zero. That is not a coincidence: starting from the mean under squared-error loss always makes the total gradient zero, since the mean is precisely the value that balances overestimates against underestimates. It also means the third term in the Gain formula above, the parent's own score, is 0 for this very first tree, which is why the numbers below come out so clean.

Step 3 — Score every candidate split. With the data sorted by income, there are four places a split could go: between A and B, between B and C, between C and D, and between D and E. Using XGBoost's own default regularization strength, lambda = 1, and gamma = 0 for this first tree:

• income < 27.5 — left {A}: GL=-35, HL=1; right {B,C,D,E}: GR=35, HR=4 → Gain = 428.75
• income < 40 — left {A,B}: GL=-55, HL=2; right {C,D,E}: GR=55, HR=3 → Gain = 882.29
• income < 52.5 — left {A,B,C}: GL=-50, HL=3; right {D,E}: GR=50, HR=2 → Gain = 729.17
• income < 75 — left {A,B,C,D}: GL=-35, HL=4; right {E}: GR=35, HR=1 → Gain = 428.75

The split at income < 40 wins clearly, more than 150 points of Gain ahead of its nearest rival. It separates the two lowest earners from the three highest earners, exactly the boundary where the actual risk scores also happen to jump.

Step 4 — Compute the optimal leaf weights. For each leaf, the correction that minimizes the regularized loss is w* = -G / (H + lambda):

• Left leaf (A, B): w* = -(-55) / (2 + 1) = 18.33
• Right leaf (C, D, E): w* = -(55) / (3 + 1) = -13.75

Step 5 — Update the predictions. That correction is not applied at full strength; it is scaled by a small learning rate, eta, so that no single tree can swing predictions too far. Using XGBoost's own default of eta = 0.3:

• A and B: 45 + 0.3 × 18.33 = 50.5
• C, D, and E: 45 + 0.3 × (-13.75) = 40.875

After a single tree, the two low-income applicants have been nudged up toward their true risk (A's true score is 80, and the prediction moved from 45 to 50.5), and the three higher earners have been nudged down (E's true score is 10, and the prediction moved from 45 to 40.875). Nobody's prediction is close to correct yet, and that is expected. One shallow tree is never meant to solve the whole problem; it takes one careful, regularized step in the right direction. The next tree computes fresh gradients against these new predictions and takes another step, and a real model repeats this hundreds of times before it is done.

LightGBM: Built for Speed at Scale

XGBoost's engineering held up well, but datasets kept growing through the mid-2010s: millions of rows, thousands of columns, features built from years of clickstream or transaction logs. Scanning every possible split point, on every feature, for every row, on every one of hundreds of trees, is an enormous amount of repeated arithmetic, and even XGBoost's optimizations started to strain under it. Microsoft's answer, released as open-source software in 2016, two years after XGBoost's own debut, was LightGBM. Its formal paper, "LightGBM: A Highly Efficient Gradient Boosting Decision Tree," by Guolin Ke and colleagues at Microsoft Research, followed the next year, at the 2017 Conference on Neural Information Processing Systems, or NeurIPS, the same pattern XGBoost had already set: software first, with the paper documenting it a year or more later.

Rather than just engineering XGBoost's exact approach harder, LightGBM changed the underlying arithmetic in three ways.

The first is histogram-based binning. Instead of treating every distinct value of a feature as a possible split point, LightGBM first buckets each continuous feature into a fixed number of bins, up to max_bin = 255 by default, and only ever evaluates splits at bin boundaries. Searching for the best split now costs time proportional to the number of bins, not the number of data points, and storing a small bin index instead of a raw floating-point number cuts memory too.

The second is GOSS, short for Gradient-based One-Side Sampling. Points with large gradients are the ones the model is still getting badly wrong, and they carry the most information about where to split next; points that are already well fit, with small gradients, add comparatively little. GOSS keeps every large-gradient point but only randomly samples a fraction of the small-gradient ones, scaling up their contribution by a fixed factor so the overall gain estimate stays statistically unbiased. Each tree ends up built from noticeably fewer rows, with almost no loss in accuracy.

The third is EFB, short for Exclusive Feature Bundling. Real tabular datasets are often sparse and high-dimensional, especially once categorical columns are one-hot encoded: a row for a customer in Mumbai has a 1 in the "city_mumbai" column and a 0 in every other city column, so those city columns are almost never nonzero at the same time. EFB detects such mutually exclusive features and bundles them into a single compound feature, shrinking the effective feature count, and the cost of building histograms over it, without losing information, since the original values can always be recovered from the bundle.

LightGBM also grows trees differently. Traditional gradient boosting, XGBoost's own default among them, grows trees level-wise: every leaf at the current depth gets split before the tree is allowed to go one level deeper, which keeps it balanced. LightGBM instead grows leaf-wise: at every step, it finds whichever single leaf anywhere in the tree would yield the largest Gain if split, and splits only that one. For a fixed leaf budget, leaf-wise growth reaches a lower loss than level-wise growth, but the resulting trees can end up narrow and deep along some branches, which risks overfitting on smaller datasets. That is why leaf-wise growth is capped by a leaf budget, LightGBM's num_leaves parameter, which defaults to 31, roughly the complexity of a balanced tree five levels deep (two to the power five, minus one, equals 31), even though a leaf-wise tree is rarely shaped like a balanced one in practice. XGBoost later added its own optional "lossguide" growth policy, which its own documentation describes as mimicking this leaf-wise behavior, though "depthwise," its original level-wise approach, remains the default. And since XGBoost's version 2.0, its own default split-search method has been "hist," the same histogram-binning idea LightGBM had built its name on. The two systems, built two years apart by different teams, have converged more than they have stayed apart.

Choosing Between Champions

In practice, picking between the two is rarely the highest-leverage decision in a project; the quality of the features usually matters more than the choice of library. A few patterns practitioners do rely on:

• LightGBM tends to train noticeably faster and lighter on memory on very large or very high-cardinality datasets, where histogram binning, GOSS, and EFB compound together.
• On smaller or noisier datasets, XGBoost's level-wise growth and heavier default regularization sometimes generalize a little more safely, since leaf-wise growth's deeper branches have more room to memorize noise when there is not much data to begin with.
• In serious competitions, it is common to train both, along with other model types, and blend their predictions, because their different growth strategies tend to make different kinds of mistakes.

Both stand in sharp contrast to what wins on images, audio, or free-form text, where deep neural networks are typically the stronger choice. On rows-and-columns data specifically, gradient-boosted trees have remained the default most practitioners reach for first: a genuinely active area of research, but one where, for now, these two champions have held their title for more than a decade.

Seeing Both Models in Code

Here is what that choice looks like in practice, on a small synthetic version of the NBFC's loan book, now with four features instead of one, and eight hundred applicants instead of five:

import numpy as np
import pandas as pd
from sklearn.model_selection import train_test_split
from sklearn.metrics import accuracy_score
from xgboost import XGBClassifier
from lightgbm import LGBMClassifier

# One row per past loan application: the tabular features a real
# NBFC loan desk would have on file, plus the known outcome.
rng = np.random.default_rng(42)
n = 800
monthly_income = rng.integers(15, 150, n)   # in thousands of rupees
credit_score = rng.integers(300, 900, n)    # CIBIL-style score
existing_loans = rng.integers(0, 5, n)      # active loans right now
age = rng.integers(21, 60, n)

risk = (-0.03 * monthly_income - 0.01 * credit_score
        + 6 * existing_loans - 0.05 * age + rng.normal(0, 8, n))
defaulted = (risk > np.percentile(risk, 75)).astype(int)

df = pd.DataFrame({"monthly_income": monthly_income, "credit_score": credit_score,
                    "existing_loans": existing_loans, "age": age, "defaulted": defaulted})

X, y = df.drop(columns="defaulted"), df["defaulted"]
X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.2, random_state=42, stratify=y)

xgb_model = XGBClassifier(n_estimators=300, max_depth=4, learning_rate=0.1,
                           reg_lambda=1.0, eval_metric="logloss", random_state=42)
xgb_model.fit(X_train, y_train)
print("XGBoost accuracy:", accuracy_score(y_test, xgb_model.predict(X_test)))

lgb_model = LGBMClassifier(n_estimators=300, num_leaves=31,
                            learning_rate=0.1, random_state=42, verbosity=-1)
lgb_model.fit(X_train, y_train)
print("LightGBM accuracy:", accuracy_score(y_test, lgb_model.predict(X_test)))

Both models are handed the identical four columns and asked to predict the same defaulted flag; everything this chapter just traced by hand, Gain calculations, leaf weights, histogram binning, GOSS sampling, happens automatically inside .fit(). Notice the learning rate here is set to 0.1, slower than XGBoost's own default of 0.3, paired with a higher tree count of 300 boosting rounds: a very common trade in practice, since smaller, more numerous steps tend to generalize better than a few large ones. Run against a held-out 20% test split, this particular dataset lands both models at essentially the same accuracy, around 78%, a useful reminder that once two well-engineered boosting libraries are given the same features, their predictive ceiling on a given dataset often converges, even though they arrive there down different internal roads.

Back to the Loan Application

The "Congratulations! You are pre-approved" message from the opening of this chapter is this same mechanism, scaled up: not one feature but dozens, credit bureau history, device fingerprint, past repayment behavior, existing EMIs, and not five past applicants but hundreds of thousands, feeding hundreds of boosting rounds rather than one. Somewhere inside that model is a Gain formula just like the one computed by hand here, deciding which feature and which threshold to split on next; gradients and Hessians standing in for "how wrong, and how sure"; a regularization penalty stopping any single tree from memorizing one unusual applicant; and, if the lender operates at real scale, histogram binning, GOSS, and EFB quietly cutting the arithmetic down so the decision lands in seconds rather than minutes. Whenever a real-world problem shows up as a spreadsheet, rows of examples, columns of features, whether the rows are loan applicants, hospital admissions, or IPL batting averages, XGBoost and LightGBM are exactly where a serious tabular-data project should start.

Think About It

Think about this: How would you explain xgboost and lightgbm: the champions of tabular data to a friend who has never seen a computer? What real-world analogy would you use? Imagine you had to build a system using these concepts — what would be your first step? Try this: before moving on, write down three things you learned and one question you still have.