Training ML Models on Imbalanced Datasets

A pedagogical study across 4 datasets: full data vs. downsampling vs. class weighting

tech
ml
Published

August 29, 2026

Keywords

imbalanced classification, downsampling, class weight, average precision, ROC-AUC, calibration, loan default, fraud detection, logistic regression, LightGBM

Imbalanced classification is one of those topics where most of us Data Scientists know the standard moves: downsample the majority or upsample the minority class, or pass class_weight='balanced'. For me, I never really took the chance to sit down and explore this topic, so this article is my attempt to understand what’s going on behind the problem of “imbalanced class”—which exists in the majority of ML problems I have been dealing with.

My plan is to build the core intuition on one dataset first: the Kaggle Give Me Some Credit loan default dataset, where about 6.7% of borrowers default. Then I’ll check whether the conclusions survive contact with three more datasets that have very different imbalance ratios and feature types.

The data: 6.7% defaults

Code 
import os
from pathlib import Path

import numpy as np
import pandas as pd
import plotly.graph_objects as go
from plotly.subplots import make_subplots
from dotenv import load_dotenv

# Registers and activates the shared `dvq` Plotly template.
import dvq_theme

# Load credentials from the repo-root .env, then map KAGGLE_API_TOKEN (KGAT_...) into KAGGLE_KEY for kagglehub.
load_dotenv(Path.cwd().parent.parent / ".env")
if os.environ.get("KAGGLE_API_TOKEN") and not os.environ.get("KAGGLE_KEY"):
    os.environ["KAGGLE_KEY"] = os.environ["KAGGLE_API_TOKEN"]

# Per-strategy colors, shared by every figure below.
# Key 1 is the strong shade (positives / series color), key 0 the soft shade (negatives).
COLOR_FULL        = {0: "#cccccc", 1: "#555555"}
COLOR_DOWNSAMPLED = {0: "#b0cfe8", 1: dvq_theme.ACCENT}
COLOR_BALANCED    = {0: "#f5dbb8", 1: "#D4863C"}

RANDOM_STATE = 42
DATA_DIR = Path("data")
DATA_DIR.mkdir(exist_ok=True)
import kagglehub

dataset_path = kagglehub.dataset_download("brycecf/give-me-some-credit-dataset")
csv_path = next(Path(dataset_path).glob("cs-training.csv"))

df = pd.read_csv(csv_path, index_col=0)
df = df.dropna(subset=["SeriousDlqin2yrs"])
df["SeriousDlqin2yrs"] = df["SeriousDlqin2yrs"].astype(int)
df["SeriousDlqin2yrs"].value_counts(normalize=True)
SeriousDlqin2yrs
0    0.93316
1    0.06684
Name: proportion, dtype: float64

A stratified train/val split

from sklearn.model_selection import train_test_split

X = df.drop(columns=["SeriousDlqin2yrs"])
y = df["SeriousDlqin2yrs"]

X_train, X_val, y_train, y_val = train_test_split(
    X, y, test_size=0.2, stratify=y, random_state=RANDOM_STATE
)

print(f"train: {len(X_train):>7,}   positives: {int(y_train.sum()):>5,}  ({y_train.mean()*100:.3f}%)")
print(f"val:   {len(X_val):>7,}   positives: {int(y_val.sum()):>5,}  ({y_val.mean()*100:.3f}%)")
train: 120,000   positives: 8,021  (6.684%)
val:    30,000   positives: 2,005  (6.683%)

Baseline: train on the data as-is

The simplest thing to do with an imbalanced dataset is to ignore the imbalance entirely: fit a model on the data exactly as it comes and see how it behaves. This’s going to be the baseline everything else gets compared against. I’m using a plain logistic regression with median imputation and robust scaling so the only thing changing between experiments is the training data, not the model.

from sklearn.linear_model import LogisticRegression
from sklearn.preprocessing import RobustScaler
from sklearn.impute import SimpleImputer
from sklearn.pipeline import Pipeline

baseline = Pipeline([
    ("imputer", SimpleImputer(strategy="median")),
    ("scaler", RobustScaler()),
    ("lr", LogisticRegression(max_iter=1000, random_state=RANDOM_STATE)),
])
baseline.fit(X_train, y_train)

scores_baseline = baseline.predict_proba(X_val)[:, 1]

Downsampling to 1:1

Downsampling is the most straight-forward way to make the class balanced: throw away negatives until the two classes are the same size. Here that means keeping all 8,021 defaults from the training set and randomly dropping all but 8,021 of the non-defaults, so the model trains on a 50/50 world instead of a 6.7% one.

pos_idx = y_train[y_train == 1].index
neg_idx = y_train[y_train == 0].index
neg_sampled = np.random.RandomState(RANDOM_STATE).choice(neg_idx, size=len(pos_idx), replace=False)
ds_idx = np.concatenate([pos_idx, neg_sampled])

X_train_ds = X_train.loc[ds_idx]
y_train_ds = y_train.loc[ds_idx]
print(f"downsampled train: {len(X_train_ds):,}  (positives: {int(y_train_ds.sum()):,})")

downsampled = Pipeline([
    ("imputer", SimpleImputer(strategy="median")),
    ("scaler", RobustScaler()),
    ("lr", LogisticRegression(max_iter=1000, random_state=RANDOM_STATE)),
])
downsampled.fit(X_train_ds, y_train_ds)

scores_downsampled = downsampled.predict_proba(X_val)[:, 1]
downsampled train: 16,042  (positives: 8,021)

Comparing the two models

Code 
from sklearn.metrics import (
    average_precision_score,
    roc_auc_score,
    precision_score,
    recall_score,
    f1_score,
)
from IPython.display import HTML

def metrics_row(name, y_true, scores, threshold=0.5):
    y_pred = (scores >= threshold).astype(int)
    return {
        "model": name,
        "AP": average_precision_score(y_true, scores),
        "ROC-AUC": roc_auc_score(y_true, scores),
        f"precision@{threshold}": precision_score(y_true, y_pred, zero_division=0),
        f"recall@{threshold}": recall_score(y_true, y_pred, zero_division=0),
        f"f1@{threshold}": f1_score(y_true, y_pred, zero_division=0),
    }

metric_names = ["AP", "ROC-AUC", "[email protected]", "[email protected]", "[email protected]"]
metric_order = list(reversed(metric_names))
model_specs = [
    ("imbalanced (full)", scores_baseline, COLOR_FULL[1]),
    ("downsampled 1:1", scores_downsampled, COLOR_DOWNSAMPLED[1]),
]

fig = go.Figure()
for name, scores, color in model_specs:
    row = metrics_row(name, y_val, scores)
    values = [row["AP"], row["ROC-AUC"], row["[email protected]"], row["[email protected]"], row["[email protected]"]]
    fig.add_trace(go.Bar(
        name=name,
        x=values,
        y=metric_names,
        orientation="h",
        marker_color=color,
        text=[f"{v:.3f}" for v in values],
        textposition="outside",
        textfont=dict(size=12),
        hovertemplate="%{y}: %{x:.4f}<extra></extra>",
    ))

fig.update_layout(
    title="Metrics at threshold 0.5",
    barmode="group",
    autosize=True,
    height=420,
    legend=dict(x=0.5, y=1.1, orientation="h", xanchor="center"),
    hoverlabel=dict(bgcolor="#1a1a1a", font_color="white", bordercolor="#1a1a1a"),
    margin=dict(t=80, l=110),
)
fig.update_xaxes(title_text="Score", range=[0, 1.15])
fig.update_yaxes(categoryorder="array", categoryarray=metric_order)

HTML(fig.to_html(
    include_plotlyjs="cdn",
    full_html=False,
    div_id="fig-metrics-bar",
    config={"responsive": True},
    default_width="100%",
    default_height="420px",
))

The full data model has higher precision at threshold 0.5 (57.6% vs. 19.6% for the downsampled model), but only because it barely makes any positive predictions. It flagged just 158 transactions out of 30,000 as likely defaults, catching 91 of the 2,005 actual defaults. That’s 4.5% recall. When it fires, it’s usually right; it almost never fires.

The downsampled model is the opposite: 1,315 defaults caught (65.6% recall) at the cost of 5,390 false alarms (19.6% precision). It’s noisier but it actually tries to find defaults.

The problem with comparing models this way is that precision, recall, and F1 are all computed at a single threshold: 0.5 here. For any other threshold, precision and recall will change. So instead of picking one, what if we measured precision and recall at every possible threshold? That’s what Rrecision-Recall curve is for. Essentially we sweep the threshold from 1 down to 0, record precision and recall at each step, and plot them against each other.

Code 
from sklearn.metrics import precision_recall_curve
from IPython.display import HTML

fig = go.Figure()

for name, scores, color in [
    ("imbalanced (full)", scores_baseline, COLOR_FULL[1]),
    ("downsampled 1:1", scores_downsampled, COLOR_DOWNSAMPLED[1]),
]:
    prec, rec, thr = precision_recall_curve(y_val, scores)
    # precision_recall_curve returns len(thr) == len(prec) - 1; pad to align
    thr_padded = np.concatenate([thr, [np.nan]])
    ap = average_precision_score(y_val, scores)
    fig.add_trace(go.Scatter(
        x=rec, y=prec,
        mode="lines",
        name=f"{name}  (AP={ap:.3f})",
        line=dict(color=color, width=2),
        customdata=thr_padded,
        hovertemplate="recall: %{x:.3f}<br>precision: %{y:.3f}<br>threshold: %{customdata:.3f}<extra></extra>",
    ))

random_precision = y_val.mean()
fig.add_hline(
    y=random_precision,
    line_dash="dot",
    line_color="#444444",
    annotation_text=f"random classifier ({random_precision:.3f})",
    annotation_position="bottom right",
    annotation_font_size=11,
)

fig.update_layout(
    title="Precision-Recall curve",
    autosize=True,
    height=420,
    xaxis=dict(title="Recall", range=[0, 1]),
    yaxis=dict(title="Precision", range=[0, 1]),
    legend=dict(x=0.45, y=0.95),
    hoverlabel=dict(bgcolor="#1a1a1a", font_color="white", bordercolor="#1a1a1a"),
)

HTML(fig.to_html(
    include_plotlyjs="cdn",
    full_html=False,
    div_id="fig-pr-curve",
    config={"responsive": True},
    default_width="100%",
    default_height="420px",
))

Each point on the curve is a threshold. Moving left to right, the threshold is dropping: you’re flagging more borrowers as likely defaults (recall goes up), but you’re also picking up more false alarms (precision goes down).1 The dotted line at 6.7% is the random classifier baseline. Picture a model whose scores carry no information about who actually defaults, in other words, independent of the label (the exact distribution doesn’t matter; uniform, Gaussian, anything works). Pick any threshold on such a model and the borrowers you flag are just a random sample of the population, so the fraction of them that actually defaults is simply the overall default rate: 6.7%. That holds at every recall level, no matter how many borrowers you flag, which is why the baseline is a flat line. It’s the bar a real model has to clearly outperform to be worth anything.

A model with a higher curve is better at every operating point: for any recall target you pick, it achieves it at higher precision. The downsampled curve sits above the full data curve across almost the entire recall range, which confirms that the gap between these two models isn’t a threshold artifact: it holds everywhere.

The area under this curve is what we want to maximize. It’s a single number that summarizes the model’s precision-recall tradeoff across all thresholds, without requiring you to commit to any one of them. Sklearn’s average_precision_score computes this as the weighted mean of precision at each step, weighted by the change in recall, numerically equivalent to the area. That’s Average Precision (AP).

Finding a positive early in the ranked list contributes more to AP than finding one late, because precision is higher near the top, so each early hit adds more area. This maps directly to how imbalanced problems actually get deployed: with a 6.7% default rate and a finite team of analysts, you’d flag roughly that top few percent of applications and never operate at a 50% flag rate. AP measures quality exactly in the region you’ll actually use, making it often an important metric for imbalanced dataset.

That said, AP has one blind spot: it’s sensitive to the positive rate in your evaluation set. In a monthly retraining setup where the holdout (e.g. 1 day of data as test set) happens to have significantly less number positives than usual, AP drops even if the model’s ranking quality is unchanged as there are fewer positives to find.2 To distinguish a real degradation from a base rate fluctuation, you need a metric that isn’t affected by the positive rate at all. That’s what the ROC curve provides.

The ROC curve comes from the same “sweep all thresholds” idea, but pairs recall with a different second axis. Instead of asking “of everyone I flagged, how many were real defaults?” (precision), it asks “of all safe borrowers, how many did I wrongly flag?”—that’s the false positive rate (FPR). The reason this makes the ROC curve insensitive to the positive rate is that both axes are now computed on their own class in isolation. TPR only looks at actual defaulters, FPR only looks at actual safe borrowers, and neither calculation involves the other class. So the ratio of positives to negatives in your evaluation set never enters the picture. To plot the ROC curve, we wweep the threshold from 1 to 0, plot recall on the y-axis and FPR on the x-axis.

Code 
from sklearn.metrics import roc_curve

fig = go.Figure()

for name, scores, color in [
    ("imbalanced (full)", scores_baseline, COLOR_FULL[1]),
    ("downsampled 1:1", scores_downsampled, COLOR_DOWNSAMPLED[1]),
]:
    fpr, tpr, thr = roc_curve(y_val, scores)
    auc = roc_auc_score(y_val, scores)
    fig.add_trace(go.Scatter(
        x=fpr, y=tpr,
        mode="lines",
        name=f"{name}  (AUC={auc:.3f})",
        line=dict(color=color, width=2),
        customdata=thr,
        hovertemplate="FPR: %{x:.3f}<br>TPR: %{y:.3f}<br>threshold: %{customdata:.3f}<extra></extra>",
    ))

fig.add_trace(go.Scatter(
    x=[0, 1], y=[0, 1],
    mode="lines",
    name="random classifier",
    line=dict(color="#444444", width=1.5, dash="dot"),
    showlegend=True,
    hoverinfo="skip",
))

fig.update_layout(
    title="ROC curve",
    autosize=True,
    height=420,
    xaxis=dict(title="False Positive Rate", range=[0, 1]),
    yaxis=dict(title="True Positive Rate (Recall)", range=[0, 1]),
    legend=dict(x=0.45, y=0.12),
    hoverlabel=dict(bgcolor="#1a1a1a", font_color="white", bordercolor="#1a1a1a"),
)

HTML(fig.to_html(
    include_plotlyjs="cdn",
    full_html=False,
    div_id="fig-roc-curve",
    config={"responsive": True},
    default_width="100%",
    default_height="420px",
))

The area under the ROC curve is ROC-AUC. Picture the curve as a staircase you build by sweeping the threshold from high to low: each time you cross a non-defaulter, FPR ticks up and you take a horizontal step; each time you cross a defaulter, TPR ticks up and you take a vertical step. Area only accumulates on horizontal steps.

When you cross a non-defaulter, the rectangle you add has width \(1/n_{\text{neg}}\) and height equal to the TPR at that moment, which is the fraction of defaulters already ranked above this non-defaulter. The area of that rectangle has a concrete interpretation: it counts how many defaulters beat this particular non-defaulter, normalized by the total number of (defaulter, non-defaulter) pairs. Sum those areas across all non-defaulters and you’ve counted every favorable pair (defaulter scored higher) exactly once. Here’s that sweep animated with 2 defaulters (0.9, 0.5) and 3 non-defaulters (0.7, 0.3, 0.1), hit Play and watch each threshold crossing:3

Code 
import plotly.graph_objects as go
from IPython.display import HTML

# Tiny example from the footnote: 2 defaulters (D), 3 non-defaulters (N)
n_pos, n_neg = 2, 3
points = [(0.9, 'D'), (0.7, 'N'), (0.5, 'D'), (0.3, 'N'), (0.1, 'N')]

DIV_ID = 'fig-roc-auc-staircase-anim'


def _hex_rgba(hex_color, alpha):
    h = hex_color.lstrip('#')
    r, g, b = int(h[:2], 16), int(h[2:4], 16), int(h[4:], 16)
    return f'rgba({r},{g},{b},{alpha})'


ACCENT = dvq_theme.ACCENT
FILL_COLOR = _hex_rgba(ACCENT, 0.12)


def build_states():
    cx, cy = [0.0], [0.0]
    fpr, tpr, nps, nns, area = 0.0, 0.0, 0, 0, 0.0
    out = [{'cx': [0.0], 'cy': [0.0], 'area': 0.0, 'active': None,
            'label': 'Threshold above all scores — curve starts at origin'}]
    for i, (score, pt) in enumerate(points):
        if pt == 'D':
            nps += 1
            tpr = nps / n_pos
            cx.append(fpr); cy.append(tpr)
            label = (f'score={score} (Defaulter)<br>'
                     f'{nps}/{n_pos} defaulters seen → TPR = {tpr:.2f}  (vertical step, +0 area)')
        else:
            nns += 1
            old_fpr, fpr = fpr, nns / n_neg
            da = (fpr - old_fpr) * tpr
            area += da
            cx.append(fpr); cy.append(tpr)
            label = (f'score={score} (Non-defaulter)<br>'
                     f'{nns}/{n_neg} non-defaulters seen → FPR = {fpr:.2f}  (area +{da:.3f}, total {area:.3f})')
        out.append({'cx': list(cx), 'cy': list(cy), 'area': area, 'active': i, 'label': label})
    return out


states = build_states()


def fill_polygon(cx, cy):
    if len(cx) <= 1:
        return [0, 0, 0], [0, 0, 0]
    return cx + [cx[-1], 0], cy + [0, 0]


def score_strip_annots(active_idx):
    annots = []
    for i, (score, pt) in enumerate(points):
        if active_idx is None or i > active_idx:
            color, text = '#555555', f'{score}({pt})'
        elif i == active_idx:
            color = ACCENT if pt == 'D' else '#cccccc'
            text = f'<b>▶ {score}({pt})</b>'
        else:
            color = ACCENT if pt == 'D' else '#999999'
            text = f'{score}({pt})'
        annots.append(dict(
            x=(i + 0.5) / len(points), y=1.32,
            xref='paper', yref='paper',
            text=text, showarrow=False,
            font=dict(size=12, color=color),
        ))
    return annots


def step_annot(label):
    return dict(
        x=0.5, y=1.07, xref='paper', yref='paper',
        text=label, showarrow=False, yanchor='bottom',
        font=dict(size=11.5), align='center',
    )


frames = []
for i, s in enumerate(states):
    fx, fy = fill_polygon(s['cx'], s['cy'])
    frames.append(go.Frame(
        name=str(i), traces=[1, 2],
        data=[
            go.Scatter(x=s['cx'], y=s['cy'], mode='lines',
                       line=dict(color=ACCENT, width=2.5)),
            go.Scatter(x=fx, y=fy, mode='none', fill='toself', fillcolor=FILL_COLOR),
        ],
        layout=go.Layout(
            annotations=score_strip_annots(s['active']) + [step_annot(s['label'])],
        ),
    ))

fx0, fy0 = fill_polygon(states[0]['cx'], states[0]['cy'])
fig = go.Figure(
    data=[
        go.Scatter(x=[0, 1], y=[0, 1], mode='lines',
                   line=dict(color='#555', dash='dot', width=1.5),
                   showlegend=False, hoverinfo='skip'),
        go.Scatter(x=states[0]['cx'], y=states[0]['cy'], mode='lines',
                   line=dict(color=ACCENT, width=2.5),
                   showlegend=False, hoverinfo='skip'),
        go.Scatter(x=fx0, y=fy0, mode='none', fill='toself', fillcolor=FILL_COLOR,
                   showlegend=False, hoverinfo='skip'),
    ],
    frames=frames,
    layout=go.Layout(
        width=510, height=560, autosize=False,  # fixed: Plotly renders at 0-width inside a closed <details>; explicit px avoids the 700px fallback
        margin=dict(t=155, b=120, l=60, r=40),
        xaxis=dict(title='False Positive Rate (FPR)', range=[-0.03, 1.03], zeroline=False),
        yaxis=dict(title='True Positive Rate (TPR)', range=[-0.03, 1.0], zeroline=False),
        annotations=score_strip_annots(None) + [step_annot(states[0]['label'])],
        updatemenus=[dict(
            type='buttons', showactive=False, direction='right',
            x=0.5, y=-0.13, xanchor='center', yanchor='top',
            buttons=[
                dict(label='▶  Play', method='animate',
                     args=[None, {'frame': {'duration': 1400, 'redraw': True},
                                  'fromcurrent': True, 'mode': 'immediate',
                                  'transition': {'duration': 500}}]),
                dict(label='⏸  Pause', method='animate',
                     args=[[None], {'frame': {'duration': 0}, 'mode': 'immediate'}]),
            ],
        )],
        sliders=[dict(
            active=0,
            steps=[dict(
                method='animate',
                args=[[str(i)], {'mode': 'immediate',
                                  'frame': {'duration': 0, 'redraw': True},
                                  'transition': {'duration': 300}}],
                label=f'Step {i}',
            ) for i in range(len(states))],
            x=0.0, y=-0.04, len=1.0,
            currentvalue=dict(prefix='', xanchor='center', font=dict(size=11)),
            pad=dict(t=70),
        )],
        hoverlabel=dict(bgcolor='#1a1a1a', font_color='white'),
    ),
)

raw_html = fig.to_html(
    include_plotlyjs='cdn',
    full_html=False,
    div_id=DIV_ID,
    config={'responsive': True},
    default_width='100%',
    default_height='560px',
)
# Plotly's to_html auto-plays all frames on load; replace with a jump to frame 0
# so the animation starts paused and waits for the reader to click Play.
raw_html = raw_html.replace(
    f"Plotly.animate('{DIV_ID}', null);",
    f"Plotly.animate('{DIV_ID}', ['0'], {{transition: {{duration: 0}}, frame: {{duration: 0, redraw: true}}}});",
)
HTML(raw_html)

That’s why ROC-AUC equals P(score(defaulter) > score(non-defaulter)) when you pick a random defaulter and a random non-defaulter from the dataset. The staircase is doing pair-counting in disguise: each non-defaulter “collects” the defaulters ranked above it, and the total area is favorable pairs divided by total pairs, exactly the probability for uniform sampling. A model that perfectly ranks all defaulters above all non-defaulters takes all its vertical steps before any horizontal ones, so every rectangle has height 1: area = 1.0. Only the relative ordering of scores matters, not their actual values.

On the other hand, a random classifier ROC AUC baseline is exactly 0.5: it ranks a positive above a negative half the time by chance, which is why it sits on the diagonal. The number has a direct probabilistic meaning regardless of class balance or score scale, and it lets you compare models fairly across datasets with different positive rates since neither TPR nor FPR depends on how many of the other class there are. That’s why ROC-AUC is the default go-to metric for classification models.

In an imbalanced setting like ours, both models look considerably better in terms of ROC AUC than on the PR curve: AUC 0.807 vs AP 0.331 for the downsampled model. This is the FPR denominator effect: with 27,995 non-defaulters, even 1,400 false alarms register as just FPR = 0.05.

Where the scores actually land

AP and ROC-AUC each squeeze the whole comparison into a single number, which is handy for comparing models but hides what they’re doing to individual predictions. Let’s look at the raw shape of the output: how the predicted scores spread out for defaulters and non-defaulters under each model.

Code 
from IPython.display import HTML

bins = np.arange(0, 1.02, 0.02)
bin_centers = (bins[:-1] + bins[1:]) / 2
row_colors = {
    "imbalanced (full)": COLOR_FULL,
    "downsampled 1:1":   COLOR_DOWNSAMPLED,
}

fig = make_subplots(
    rows=2, cols=2,
    subplot_titles=(
        "negatives (class 0)", "positives (class 1)",
        "negatives (class 0)", "positives (class 1)",
    ),
    row_titles=["imbalanced (full)", "downsampled 1:1"],
    shared_yaxes="all",
    shared_xaxes="all",
    vertical_spacing=0.14,
    horizontal_spacing=0.1,
)

for row, (name, scores) in enumerate(
    [("imbalanced (full)", scores_baseline), ("downsampled 1:1", scores_downsampled)],
    start=1,
):
    for col, label in enumerate([0, 1], start=1):
        mask = y_val == label
        counts, _ = np.histogram(scores[mask], bins=bins)
        pct = counts / counts.sum() * 100
        fig.add_trace(
            go.Bar(
                x=bin_centers,
                y=pct,
                width=0.02,
                customdata=counts,
                hovertemplate="score: %{x:.2f}<br>%{y:.1f}%<br>count: %{customdata:,d}<extra></extra>",
                marker=dict(color=row_colors[name][label], line=dict(width=0)),
                showlegend=False,
            ),
            row=row, col=col,
        )

        n_above = int((scores[mask] >= 0.5).sum())
        fig.add_vline(
            x=0.5, row=row, col=col,
            line_dash="dash", line_color="#444444", line_width=1.5,
        )
        fig.add_annotation(
            x=0.52, y=88,
            text=f"{n_above:,} above 0.5",
            row=row, col=col,
            showarrow=False,
            xanchor="left",
            font=dict(size=11, color="#444444"),
        )

fig.update_layout(
    height=560,
    autosize=True,
    title="Score distributions by class — % within class (hover for raw count)",
    bargap=0,
    hoverlabel=dict(bgcolor="#1a1a1a", font_color="white", bordercolor="#1a1a1a"),
)
fig.update_xaxes(title_text="", range=[0, 1])
fig.update_xaxes(title_text="predicted P(class=1)", row=2)
fig.update_yaxes(title_text="% of class", range=[0, 100])

HTML(fig.to_html(
    include_plotlyjs="cdn",
    full_html=False,
    div_id="fig-scores-2x2",
    config={"responsive": True},
    default_width="100%",
    default_height="560px",
))

The full data model’s positive side (top-right) shows just how rarely it fires: it predicted “probably not a default” for nearly every borrower who ended up defaulting, with only a handful of true positives crossing the line.

The downsampled model shifts the positive distribution sharply upward, catching most of the actual defaults now scoring above 0.5. But look at the negative side (bottom-left): a real chunk of non-defaulters cross the threshold too.

So it’s becoming clear that the full data model avoided most of those false alarms mostly because its scores are anchored near zero. Our two models are just calibrated to different operating points.

Why does training on downsampled data push scores higher?

The distributions just showed the downsampled model’s scores sitting higher across the board, not just ranked differently. That shift is worth pausing on, because it falls straight out of how logistic regression is trained. Logistic regression models the probability of a positive outcome as:

\[p = \text{sigmoid}(w \cdot x + b) = \frac{1}{1 + e^{-(w \cdot x + b)}}\]

The weights \(w\) learn which features push the probability up or down. The bias \(b\) is a constant added to every sample’s raw score before it gets squashed through sigmoid, the model’s baseline suspicion before it looks at any features.

The model is trained to minimize cross-entropy loss:

\[L = -\sum_i \left[ y_i \log(p_i) + (1 - y_i) \log(1 - p_i) \right]\]

The \(\log\) terms are what give this its character: \(\log(p)\) shoots toward \(-\infty\) as \(p \to 0\), so being confidently wrong incurs a catastrophically large penalty. Think of it as a “how surprised were you?” scorer. Predict 99% default and it turns out to be a default—barely surprised, tiny penalty. Predict 1% default and it turns out to be a default—extremely surprised, huge penalty. The model’s entire job during training is to stop being surprised.

Now zoom in on the bias. If it’s set too low, the model is systematically shocked every time a default shows up. If it’s too high, it’s systematically shocked by the ~93% of borrowers who don’t default. The only resting point where the bias stops accumulating surprise from both sides is when it matches how often defaults actually occur in training, the training positive rate. Any other value keeps bleeding into the loss.

This is why the downsampled model’s scores are inflated. It was trained to stop being surprised in a world where defaults are 50% common. When deployed into a world where defaults are 6.7% common, its baseline suspicion is miscalibrated, so scores shift up.

What happens when we use class_weight='balanced'?

The other textbook fix for an imbalanced dataset is class_weight='balanced'. Instead of throwing data away, it keeps everything and multiplies each sample’s contribution to the loss by a class-specific weight. Sklearn computes that weight as n_samples / (n_classes × class_count), roughly 7.5× for defaults and 0.54× for non-defaulters in our dataset.

This is appealing because you don’t throw away data. However, with both classes weighted equally, the bias settles at the same resting point as the downsampled model, tuned to the 50/50 world the loss sees rather than the real 6.7% base rate. Scores will still be inflated.

The genuine difference is in the feature weights \(w\). The downsampled model trained on 16,042 examples; the balanced model trains on all 120k. More diverse non-defaulters should give the model a richer picture of the decision boundary. Let’s look at the data to see whether that actually helps.

Code 
balanced = Pipeline([
    ("imputer", SimpleImputer(strategy="median")),
    ("scaler", RobustScaler()),
    ("lr", LogisticRegression(max_iter=1000, random_state=RANDOM_STATE, class_weight="balanced")),
])
balanced.fit(X_train, y_train)

scores_balanced = balanced.predict_proba(X_val)[:, 1]
Code 
from IPython.display import HTML

metric_names = ["AP", "ROC-AUC", "[email protected]", "[email protected]", "[email protected]"]
metric_order = list(reversed(metric_names))
model_specs_3 = [
    ("imbalanced (full)", scores_baseline, COLOR_FULL[1]),
    ("downsampled 1:1", scores_downsampled, COLOR_DOWNSAMPLED[1]),
    ("balanced weights", scores_balanced, COLOR_BALANCED[1]),
]

fig = go.Figure()
for name, scores, color in model_specs_3:
    row = metrics_row(name, y_val, scores)
    values = [row["AP"], row["ROC-AUC"], row["[email protected]"], row["[email protected]"], row["[email protected]"]]
    fig.add_trace(go.Bar(
        name=name,
        x=values,
        y=metric_names,
        orientation="h",
        marker_color=color,
        text=[f"{v:.3f}" for v in values],
        textposition="outside",
        textfont=dict(size=12),
        hovertemplate="%{y}: %{x:.4f}<extra></extra>",
    ))

fig.update_layout(
    title="Metrics at threshold 0.5",
    barmode="group",
    bargap=0.25,
    autosize=True,
    height=480,
    legend=dict(x=0.5, y=1.1, orientation="h", xanchor="center"),
    hoverlabel=dict(bgcolor="#1a1a1a", font_color="white", bordercolor="#1a1a1a"),
    margin=dict(t=80, l=110),
)
fig.update_xaxes(title_text="Score", range=[0, 1.15])
fig.update_yaxes(categoryorder="array", categoryarray=metric_order)

HTML(fig.to_html(
    include_plotlyjs="cdn",
    full_html=False,
    div_id="fig-metrics-bar-3models",
    config={"responsive": True},
    default_width="100%",
    default_height="480px",
))
Code 
bins = np.arange(0, 1.02, 0.02)
bin_centers = (bins[:-1] + bins[1:]) / 2
row_colors = {
    "imbalanced (full)": COLOR_FULL,
    "downsampled 1:1":   COLOR_DOWNSAMPLED,
    "balanced weights":  COLOR_BALANCED,
}

fig = make_subplots(
    rows=3, cols=2,
    subplot_titles=(
        "negatives (class 0)", "positives (class 1)",
        "negatives (class 0)", "positives (class 1)",
        "negatives (class 0)", "positives (class 1)",
    ),
    row_titles=["imbalanced (full)", "downsampled 1:1", "balanced weights"],
    shared_yaxes="all",
    shared_xaxes="all",
    vertical_spacing=0.10,
    horizontal_spacing=0.1,
)

for row, (name, scores) in enumerate(
    [
        ("imbalanced (full)", scores_baseline),
        ("downsampled 1:1", scores_downsampled),
        ("balanced weights", scores_balanced),
    ],
    start=1,
):
    for col, label in enumerate([0, 1], start=1):
        mask = y_val == label
        counts, _ = np.histogram(scores[mask], bins=bins)
        pct = counts / counts.sum() * 100
        fig.add_trace(
            go.Bar(
                x=bin_centers,
                y=pct,
                width=0.02,
                customdata=counts,
                hovertemplate="score: %{x:.2f}<br>%{y:.1f}%<br>count: %{customdata:,d}<extra></extra>",
                marker=dict(color=row_colors[name][label], line=dict(width=0)),
                showlegend=False,
            ),
            row=row, col=col,
        )

        n_above = int((scores[mask] >= 0.5).sum())
        fig.add_vline(
            x=0.5, row=row, col=col,
            line_dash="dash", line_color="#444444", line_width=1.5,
        )
        fig.add_annotation(
            x=0.52, y=88,
            text=f"{n_above:,} above 0.5",
            row=row, col=col,
            showarrow=False,
            xanchor="left",
            font=dict(size=11, color="#444444"),
        )

fig.update_layout(
    height=560,
    autosize=True,
    title="Score distributions by class — % within class (hover for raw count)",
    bargap=0,
    hoverlabel=dict(bgcolor="#1a1a1a", font_color="white", bordercolor="#1a1a1a"),
)
fig.update_xaxes(title_text="", range=[0, 1])
fig.update_xaxes(title_text="predicted P(class=1)", row=3)
fig.update_yaxes(title_text="% of class", range=[0, 100])

HTML(fig.to_html(
    include_plotlyjs="cdn",
    full_html=False,
    div_id="fig-scores-3x2",
    config={"responsive": True},
    default_width="100%",
    default_height="560px",
))

Both downsampling and balanced weights outperform the full data model on every ranking metric, which is itself the finding worth dwelling on. The usual expectation is that more data wins. Here it doesn’t, at least for LR.

Balanced weights (AP 0.323) and downsampled (AP 0.331) land in roughly the same range; the gap is small enough that neither method clearly dominates on this single split. The 120k training set with weighted loss didn’t extract meaningfully more signal than the 16k balanced subset. The score range change applies equally to both (they were optimized toward an effective 50/50 world), and the richer diversity of non-defaulters in the weighted model only moved the needle by 0.008 AP.

LightGBM: does the same story hold?

Everything so far has used logistic regression. But in generall we don’t really use LR so much for tabular dataset nowadays, as it has been the norm where gradient boosting trees methods often outperform LR with right off the bat. So I set out to rerun the same three experiments using LightGBM to see whether the same techniques behave the same way. As its equivalent of class_weight='balanced' is scale_pos_weight, setting this to n_neg / n_pos (~14 here) makes the total loss contribution from positives equal to negatives.

The common best practice when training LGBM is to use a validation data set for early stopping, preventing the model from overfitting the training dataset. So at first I think comparing LR without early stopping to LGBM with early stopping isn’t as unfair as it sounds. Turns out LR and LGBM overfit in different ways. LR optimizes a globally convex objective, so running the solver longer just brings you closer to the exact minimum of the regularized loss; more iterations don’t add capacity. Its counterpart to early stopping is the regularization strength C, and the default C=1.0 is already capping weight magnitude. LGBM works differently: it builds trees sequentially, and each tree can memorize the residuals of the previous ones. Capacity grows with tree count, so there’s a real risk of overfitting if training runs too long. Therefore we use early stopping for LGBM, with 20% of training data as a validation set.

Code 
import lightgbm as lgb

n_pos = int(y_train.sum())
n_neg = int((y_train == 0).sum())

# Carve out 20% of training data for early stopping (stratified so it has ~79 positives).
# X_val is kept fully held-out for final evaluation.
X_tr, X_es, y_tr, y_es = train_test_split(
    X_train, y_train, test_size=0.2, stratify=y_train, random_state=RANDOM_STATE
)
print(f"LGBM training set:       {len(X_tr):,}  ({int(y_tr.sum())} positives)")
print(f"Early stopping set:      {len(X_es):,}  ({int(y_es.sum())} positives)")

def ap_metric(y_true, y_pred):
    return "ap", average_precision_score(y_true, y_pred), True  # higher is better

callbacks = [lgb.early_stopping(50, verbose=False), lgb.log_evaluation(period=-1)]

def fit_lgbm(X_fit, y_fit, **kwargs):
    m = lgb.LGBMClassifier(
        n_estimators=2000, random_state=RANDOM_STATE, verbose=-1,
        num_threads=1,   # deterministic: parallel threading changes split order
        metric="None",  # suppress default binary_logloss so early stopping tracks AP only
        **kwargs
    )
    m.fit(X_fit, y_fit, eval_set=[(X_es, y_es)], eval_metric=ap_metric, callbacks=callbacks)
    return m

# Downsample from X_tr only (not X_train) so the downsampled training set
# can't include rows also used as the early-stopping holdout — otherwise
# the model would be "early stopped" against data it partly trained on.
pos_idx_lgb = y_tr[y_tr == 1].index
neg_idx_lgb = y_tr[y_tr == 0].index
neg_sampled_lgb = np.random.RandomState(RANDOM_STATE).choice(neg_idx_lgb, size=len(pos_idx_lgb), replace=False)
ds_idx_lgb = np.concatenate([pos_idx_lgb, neg_sampled_lgb])
X_tr_ds, y_tr_ds = X_tr.loc[ds_idx_lgb], y_tr.loc[ds_idx_lgb]

lgbm_baseline    = fit_lgbm(X_tr, y_tr)
lgbm_balanced    = fit_lgbm(X_tr, y_tr, scale_pos_weight=n_neg / n_pos)
lgbm_downsampled = fit_lgbm(X_tr_ds, y_tr_ds)

scores_lgbm_baseline    = lgbm_baseline.predict_proba(X_val)[:, 1]
scores_lgbm_balanced    = lgbm_balanced.predict_proba(X_val)[:, 1]
scores_lgbm_downsampled = lgbm_downsampled.predict_proba(X_val)[:, 1]

print(f"\nscale_pos_weight: {n_neg / n_pos:.1f}×")
for name, m in [("baseline", lgbm_baseline), ("scale_pos_weight", lgbm_balanced), ("downsampled", lgbm_downsampled)]:
    print(f"  LGBM {name:18s}  best_iter={m.best_iteration_}")
LGBM training set:       96,000  (6417 positives)
Early stopping set:      24,000  (1604 positives)

scale_pos_weight: 14.0×
  LGBM baseline            best_iter=41
  LGBM scale_pos_weight    best_iter=93
  LGBM downsampled         best_iter=52
Code 
strategies = [
    ("imbalanced (full)", "full", COLOR_FULL[1]),
    ("downsampled 1:1", "downsampled", COLOR_DOWNSAMPLED[1]),
    ("balanced weights", "balanced", COLOR_BALANCED[1]),
]
model_scores = {
    ("LR", "full"): scores_baseline,
    ("LR", "downsampled"): scores_downsampled,
    ("LR", "balanced"): scores_balanced,
    ("LGBM", "full"): scores_lgbm_baseline,
    ("LGBM", "downsampled"): scores_lgbm_downsampled,
    ("LGBM", "balanced"): scores_lgbm_balanced,
}
# Top-to-bottom reading order; Plotly renders a category array bottom-up.
metric_order = list(reversed(metric_names))

fig = make_subplots(rows=2, cols=1, subplot_titles=("LR", "LGBM"), vertical_spacing=0.18)

for row, model in enumerate(["LR", "LGBM"], start=1):
    for name, strat, color in strategies:
        row_metrics = metrics_row(name, y_val, model_scores[(model, strat)])
        values = [row_metrics["AP"], row_metrics["ROC-AUC"], row_metrics["[email protected]"], row_metrics["[email protected]"], row_metrics["[email protected]"]]
        fig.add_trace(go.Bar(
            name=name, x=values, y=metric_names,
            orientation="h",
            marker_color=color,
            text=[f"{v:.3f}" for v in values], textposition="outside",
            textfont=dict(size=12),
            hovertemplate="%{y}: %{x:.4f}<extra></extra>",
            showlegend=(row == 1),
            legendgroup=name,
        ), row=row, col=1)

fig.update_layout(
    title="Metrics at threshold 0.5 — LR vs LGBM",
    barmode="group",
    bargap=0.25,
    autosize=True,
    height=900,
    legend=dict(x=0.5, y=-0.06, orientation="h", xanchor="center"),
    hoverlabel=dict(bgcolor="#1a1a1a", font_color="white", bordercolor="#1a1a1a"),
    margin=dict(t=70, b=90, l=110),
)
fig.update_xaxes(title_text="Score", range=[0, 1.2])
fig.update_yaxes(categoryorder="array", categoryarray=metric_order)

HTML(fig.to_html(
    include_plotlyjs="cdn",
    full_html=False,
    div_id="fig-metrics-bar-lgbm",
    config={"responsive": True},
    default_width="100%",
    default_height="900px",
))
Code 
row_colors = {
    "LGBM — imbalanced (full)": COLOR_FULL,
    "LGBM — downsampled 1:1":   COLOR_DOWNSAMPLED,
    "LGBM — balanced weights":  COLOR_BALANCED,
}

fig = make_subplots(
    rows=3, cols=2,
    subplot_titles=(
        "negatives (class 0)", "positives (class 1)",
        "negatives (class 0)", "positives (class 1)",
        "negatives (class 0)", "positives (class 1)",
    ),
    row_titles=["imbalanced (full)", "downsampled 1:1", "balanced weights"],
    shared_yaxes="all",
    shared_xaxes="all",
    vertical_spacing=0.10,
    horizontal_spacing=0.1,
)

for row, (name, scores) in enumerate(
    [
        ("LGBM — imbalanced (full)", scores_lgbm_baseline),
        ("LGBM — downsampled 1:1", scores_lgbm_downsampled),
        ("LGBM — balanced weights", scores_lgbm_balanced),
    ],
    start=1,
):
    for col, label in enumerate([0, 1], start=1):
        mask = y_val == label
        counts, _ = np.histogram(scores[mask], bins=bins)
        pct = counts / counts.sum() * 100
        fig.add_trace(
            go.Bar(
                x=bin_centers,
                y=pct,
                width=0.02,
                customdata=counts,
                hovertemplate="score: %{x:.2f}<br>%{y:.1f}%<br>count: %{customdata:,d}<extra></extra>",
                marker=dict(color=row_colors[name][label], line=dict(width=0)),
                showlegend=False,
            ),
            row=row, col=col,
        )

        n_above = int((scores[mask] >= 0.5).sum())
        fig.add_vline(
            x=0.5, row=row, col=col,
            line_dash="dash", line_color="#444444", line_width=1.5,
        )
        fig.add_annotation(
            x=0.52, y=88,
            text=f"{n_above:,} above 0.5",
            row=row, col=col,
            showarrow=False,
            xanchor="left",
            font=dict(size=11, color="#444444"),
        )

fig.update_layout(
    height=560,
    autosize=True,
    title="LightGBM score distributions by class — % within class (hover for raw count)",
    bargap=0,
    hoverlabel=dict(bgcolor="#1a1a1a", font_color="white", bordercolor="#1a1a1a"),
)
fig.update_xaxes(title_text="", range=[0, 1])
fig.update_xaxes(title_text="predicted P(class=1)", row=3)
fig.update_yaxes(title_text="% of class", range=[0, 100])

HTML(fig.to_html(
    include_plotlyjs="cdn",
    full_html=False,
    div_id="fig-scores-lgbm-3x2",
    config={"responsive": True},
    default_width="100%",
    default_height="560px",
))

Unlike LR, LGBM’s full-data baseline already leads on both ranking metrics here — downsampling and balanced weights actually pull AP down, not up. The score distributions back that up: LGBM’s baseline scores aren’t anchored near zero the way LR’s were, so rebalancing doesn’t have the same corrective work to do.

To mitigate noise that can happen with inherent randomness, from here on results are 5-fold stratified CV (mean ± std across folds) instead of a single 80/20. Gaps smaller than roughly 2× their own std count as noise.

import experiments as ex

results_gmsc = ex.run_dataset("GMSC", DATA_DIR, n_splits=5)

The slope chart below tracks each strategy across the two model families, LR and LGBM side by side.

Code 
gmsc_stats = (
    results_gmsc
    .groupby(["model", "strategy"])["AP"]
    .agg(["mean", "std"])
)

def v(model, strategy):
    row = gmsc_stats.loc[(model, strategy)]
    return row["mean"], row["std"]

strategies = [
    ("imbalanced (full)",           v("LR", "full"),        v("LGBM", "full"),        COLOR_FULL[1]),
    ("balanced / scale_pos_weight", v("LR", "balanced"),    v("LGBM", "balanced"),    COLOR_BALANCED[1]),
    ("downsampled 1:1",             v("LR", "downsampled"), v("LGBM", "downsampled"), COLOR_DOWNSAMPLED[1]),
]

def nudge(values, min_gap=0.016):
    idx = sorted(range(len(values)), key=lambda i: values[i])
    adj = list(values)
    for i in range(1, len(idx)):
        lo, hi = idx[i - 1], idx[i]
        if adj[hi] - adj[lo] < min_gap:
            adj[hi] = adj[lo] + min_gap
    return adj

x_lr, x_lgbm, pad = 0, 1, 0.34

lr_vals   = [s[1][0] for s in strategies]
lgbm_vals = [s[2][0] for s in strategies]
lr_ys   = nudge(lr_vals)
lgbm_ys = nudge(lgbm_vals)

fig = go.Figure()

for i, (name, (lr_m, lr_s), (lgbm_m, lgbm_s), color) in enumerate(strategies):
    fig.add_trace(go.Scatter(
        x=[x_lr, x_lgbm], y=[lr_m, lgbm_m],
        mode="lines+markers", name=name,
        line=dict(color=color, width=2.5),
        marker=dict(size=10, color=color),
        error_y=dict(type="data", array=[lr_s, lgbm_s], color=color, thickness=1.5, width=5),
        legendgroup=name,
        hovertemplate=f"<b>{name}</b><br>%{{x}}: AP=%{{y:.3f}}<extra></extra>",
    ))
    fig.add_annotation(
        x=x_lr - 0.05, y=lr_ys[i], text=f"{lr_m:.3f} ± {lr_s:.3f}",
        xanchor="right", showarrow=False, font=dict(size=12, color=color),
    )
    fig.add_annotation(
        x=x_lgbm + 0.05, y=lgbm_ys[i], text=f"{lgbm_m:.3f} ± {lgbm_s:.3f}",
        xanchor="left", showarrow=False, font=dict(size=12, color=color),
    )

fig.update_xaxes(
    tickvals=[x_lr, x_lgbm], ticktext=["LR", "LGBM"],
    range=[x_lr - pad, x_lgbm + pad],
    showgrid=False, zeroline=False, tickfont=dict(size=14),
)
fig.update_yaxes(
    title_text="Average Precision (AP)", range=[0.20, 0.44],
    showgrid=True, gridcolor="rgba(255,255,255,0.06)", zeroline=False,
)

fig.update_layout(
    title="Average Precision by strategy — LR vs LGBM (mean ± std across 5 folds)",
    autosize=True,
    height=400,
    legend=dict(x=0.5, y=-0.18, orientation="h", xanchor="center"),
    hoverlabel=dict(bgcolor="#1a1a1a", font_color="white", bordercolor="#1a1a1a"),
    margin=dict(l=60, r=60, t=70, b=100),
)

HTML(fig.to_html(
    include_plotlyjs="cdn",
    full_html=False,
    div_id="fig-slope-ap-roc",
    config={"responsive": True},
    default_width="100%",
    default_height="400px",
))

Read left to right: downsampling and balanced weighting each pull LR’s dot up, but not so for LGBM.

Why the two models diverge

From theoretical standpoint I believe the reason comes down to what class imbalance actually does to each model during training.

For LR, the problem is gradient asymmetry. With most examples negative, most of the gradient signal says “push scores down.” The bias term—the model’s baseline suspicion before it looks at any features—settles at the training positive rate. The weights learn to separate classes in a skewed regime, and the decision boundary ends up in a suboptimal location. Downsampling and balanced weighting both fix this by making the gradient symmetric: the model sees equal pressure from both classes, finds a better hyperplane, and ranks more accurately.

For LGBM, that asymmetry problem doesn’t bite the same way because of how boosting works sequentially. Each tree fits the residuals of the ensemble so far. A positive example the model gets wrong has a large residual; a negative example it already handles well has a small one. The algorithm naturally concentrates attention on hard cases across boosting rounds, and in an imbalanced dataset, the hard cases are disproportionately the minority class. LGBM already does implicitly some of what downsampling and scale_pos_weight try to do explicitly—which is why reweighting tends not to add much: the positives are already getting extra attention through the residual mechanism. Downsampling is the technique more likely to cost LGBM since it also throws away most of the training data, leaving the model with a smaller, less diverse picture of the feature space to build trees on.

Do these findings apply on other datasets?

It’s also worth widening the check beyond one dataset: everything so far comes from GMSC, with one particular imbalance ratio (6.7%) and one particular kind of feature (raw, un-engineered financial ratios). If “downsampling and balanced weights help LR but hurt LGBM” is really about how these two model families handle imbalance, it should hold up on a dataset with a very different base rate and very different features too—not just this one.

So three more datasets join the study: the Kaggle Credit Card Fraud dataset (284,807 transactions, 0.17% fraud—a much more severe imbalance, and features that are already PCA components, pre-engineered for separability), the UCI Adult Census Income dataset (48,842 people, ~24% earning over $50k—mild imbalance, raw mixed categorical/numeric features), and the Sparkov simulated card-fraud dataset (300,000 transactions sampled down from 1.85M, 0.58% fraud—a severe imbalance close to Credit Card Fraud’s, but with raw features: merchant category, transaction amount, cardholder demographics, location). We re-run our pipeline of identical six conditions (LR/LGBM × full/downsampled/balanced), same preprocessing shape, evaluated with 5-fold stratified CV on each.

A note on the Sparkov dataset: it earns its spot specifically to pull apart two things that travel together in Credit Card Fraud: severe imbalance and pre-engineered features. If LR’s GMSC result was really about how separable the features already are, Sparkov’s raw features should behave like GMSC’s. If it’s actually about how severe the imbalance is, Sparkov should behave like Fraud instead, despite having nothing in common with it feature-wise.

results = pd.concat(
    [results_gmsc] + [ex.run_dataset(name, DATA_DIR, n_splits=5) for name in ["Fraud", "Adult", "Sparkov"]],
    ignore_index=True,
)
Code 
def paired_deltas(df, dataset, metric):
    """Per-fold (variant - full), then mean/std across folds — matched pairs, so
    split-to-split noise that hits every condition equally cancels out."""
    p = df[df["dataset"] == dataset].pivot_table(index=["model", "fold"], columns="strategy", values=metric)
    out = pd.DataFrame({
        "downsampled - full": p["downsampled"] - p["full"],
        "balanced - full": p["balanced"] - p["full"],
    })
    return out.groupby("model").agg(["mean", "std"]).round(4)
Code 
datasets_order = ["GMSC", "Fraud", "Adult", "Sparkov"]
models = ["LR", "LGBM"]
metrics = ["AP", "ROC-AUC"]
strat_colors = {"downsampled": COLOR_DOWNSAMPLED[1], "balanced": COLOR_BALANCED[1]}

delta_rows = []
for ds in datasets_order:
    for metric in metrics:
        d = paired_deltas(results, ds, metric)
        for model in models:
            for col, strat in [("downsampled - full", "downsampled"), ("balanced - full", "balanced")]:
                delta_rows.append({
                    "dataset": ds, "model": model, "metric": metric, "strategy": strat,
                    "mean": d.loc[model, (col, "mean")],
                    "std": d.loc[model, (col, "std")],
                })
deltas = pd.DataFrame(delta_rows)

fig = make_subplots(
    rows=2, cols=2,
    subplot_titles=["LR", "LGBM", "LR", "LGBM"],
    row_titles=["Δ AP", "Δ ROC-AUC"],
    vertical_spacing=0.18, horizontal_spacing=0.1,
)

# One y-range for all four panels: LR vs LGBM and AP vs ROC-AUC are all read
# against the same scale, so the flat ROC-AUC row is a finding, not a rendering.
# Extra headroom on top makes room for the best_iteration_ labels below.
lo = (deltas["mean"] - deltas["std"]).min()
hi = (deltas["mean"] + deltas["std"]).max()
pad = 0.08 * (hi - lo)
shared_range = [lo - pad, hi + pad * 4.2]

# Mean best_iteration_ per dataset/strategy, LGBM only — LR isn't early
# stopped (its objective is convex, so more iterations don't add capacity),
# so it has no equivalent number to annotate.
iter_stats = results[results["model"] == "LGBM"].groupby(["dataset", "strategy"])["best_iter"].mean()

for r, metric in enumerate(metrics, start=1):
    for c, model in enumerate(models, start=1):
        fig.update_yaxes(range=shared_range, dtick=0.2, row=r, col=c)
        sub = deltas[(deltas["metric"] == metric) & (deltas["model"] == model)]
        annotate_iters = model == "LGBM" and metric == "AP"
        for strat in ["downsampled", "balanced"]:
            row = sub[sub["strategy"] == strat].set_index("dataset").loc[datasets_order]
            fig.add_trace(go.Bar(
                x=datasets_order, y=row["mean"],
                error_y=dict(type="data", array=row["std"]),
                name=strat, marker_color=strat_colors[strat],
                showlegend=(r == 1 and c == 1),
                legendgroup=strat,
                hovertemplate="%{x}: %{y:.4f}<extra></extra>",
            ), row=r, col=c)
        if annotate_iters:
            # One combined label per dataset (both strategies' iteration
            # counts together), staggered across two heights so adjacent
            # categories don't compete for the same horizontal strip.
            for i, ds in enumerate(datasets_order):
                d_iters = iter_stats[(ds, "downsampled")]
                b_iters = iter_stats[(ds, "balanced")]
                y_frac = 0.86 if i % 2 == 0 else 0.55
                fig.add_annotation(
                    x=ds, y=shared_range[1] * y_frac,
                    text=f"<span style='color:{COLOR_DOWNSAMPLED[1]}'>{d_iters:.0f}</span> / "
                         f"<span style='color:{COLOR_BALANCED[1]}'>{b_iters:.0f}</span> iters",
                    showarrow=False, font=dict(size=11, color="#888888"),
                    row=r, col=c,
                )
        fig.add_hline(y=0, row=r, col=c, line_color="#444444", line_width=1)

fig.update_layout(
    title=(
        "Δ vs. full-data baseline, mean ± std across 5 folds (below zero = rebalancing hurt)"
        "<br><sup>LGBM panel labeled with mean best_iteration_ (downsampled / balanced) — LR isn't early-stopped, so it has no equivalent</sup>"
    ),
    barmode="group",
    autosize=True,
    height=580,
    legend=dict(x=0.5, y=-0.1, orientation="h", xanchor="center"),
    hoverlabel=dict(bgcolor="#1a1a1a", font_color="white", bordercolor="#1a1a1a"),
    margin=dict(t=100, b=90),
)

HTML(fig.to_html(
    include_plotlyjs="cdn",
    full_html=False,
    div_id="fig-delta-grid",
    config={"responsive": True},
    default_width="100%",
    default_height="580px",
))

There seems to be some noticeable patterns.

Downsampling never clearly helps LGBM, and on two of the four datasets it measurably hurts. On GMSC and Adult, the downsampled model’s AP loss clears the noise bar: a shrunk training set just offers less signal than the full one. On Fraud and Sparkov the picture is noisier: both deltas fall inside the ±2σ band, so downsampling’s effect there is statistically a coin flip, not a real gain or loss.4 There’s still a real wrinkle on Sparkov worth noting: the downsampled model runs for far more boosting rounds on average than the full-data one (about 97 vs. 28) before early stopping kicks in, so raw features genuinely do take more trees to make sense of. It’s just that those extra trees don’t translate into a better ranking. More trees isn’t automatically more signal.

Balanced weighting for LGBM breaks down once the minority class gets truly scarce. On GMSC and Adult, where the imbalance is moderate, it barely changes anything, a rounding error either way. On Fraud and Sparkov, where positives are rare, it’s a disaster: on Fraud, the model loses more than three-quarters of its AP score. The loss gets pushed so hard toward the tiny minority that the model overcorrects in its first few trees, mistaking a lot of ordinary cases for suspicious ones. Early stopping notices right away and shuts training down almost before it starts, a handful of trees instead of the dozens it would otherwise grow.5 Seeing the same collapse on two datasets built in completely different ways is what convinces me this is a real failure mode of the technique, not a quirk of one dataset.

ROC-AUC barely notices any of this. Every AP swing above, big or small, moves ROC-AUC by only a sliver in comparison; even Fraud’s catastrophic balanced-weighting collapse barely dents it. This is because the two metrics are weighing different things: ROC-AUC averages over every possible pair of a positive and a negative example, so a modeling change has to reorder a lot of them before the number moves. AP focuses at the top of the ranked list, where a handful of predictions decide the score, so polluting that top handful with a few false positives craters it even while the rest of the ranking barely changed.

Whether rebalancing helps LR comes down to how severe the imbalance is, not how the features were built. It clearly helps on GMSC, where the imbalance is real but not extreme (worth about seven points of AP for balanced weighting alone). It barely matters on Adult, where the imbalance was mild to begin with. And it actively backfires on both severely imbalanced datasets, Fraud and Sparkov, even though their features couldn’t be more different: one carefully engineered, one completely raw. On Fraud it costs almost 30 points of AP. That agreement is what ruled out my first guess that this was about feature quality. What Fraud and Sparkov actually share is a shortage of real examples: a few hundred to a couple thousand positives per training fold, versus thousands more on GMSC and Adult. Too few for the model to learn a stable correction from, so rebalancing ends up chasing noise instead of fixing anything real.

Closing thoughts

Personally, while this may sound obvious to others, the biggest take-away for me is that correcting for imbalance has a big impact on the score distribution. Ranking metrics won’t warn you about this: AP and ROC-AUC are invariant to any monotonic rescaling of the scores, and that rescaling is exactly what rebalancing does. So a model that looks strictly better can silently invalidate a threshold someone downstream calibrated against the old scale. If you’re on the hook for scores that stay stable and reliable across retrains, that’s a cost worth pricing in alongside whatever the technique does for your metrics. This normally leads to another practical topic: score calibration—which is out of scope for this post by definitely an area requiring the DS to think carefully about.

Coming back to the question I opened with: does rebalancing actually work? At least from this study, it really depends on how severe the imbalance is. LR only benefits from downsampling or balanced weights where the positive rate is still not so bad. Below that band, on Fraud and Sparkov, the same techniques make LR worse, and it didn’t matter that one dataset’s features were PCA components and the other’s were raw. LGBM shows a partial mirror image: balanced weighting collapses in that same severe zone, while downsampling never clears the noise bar in either direction.


If you find this article helpful, please cite this writeup as:

Quy, Dinh. (Aug 2026). Training ML Models on Imbalanced Datasets. dvquys.com. https://dvquys.com/projects/imbalanced-ml/.

Footnotes

  1. Both curves show a brief ascending portion at the far left, where precision increases as recall increases. This happens because sklearn’s precision_recall_curve computes precision and recall at each distinct score value, not one sample at a time. At the highest thresholds, you’re only flagging a handful of predictions, and each time the threshold drops into a batch of samples that’s majority positive, both recall and precision tick up together. The descent begins once you’ve exhausted that high-confidence positive cluster and start pulling in noisier, mixed-score territory.↩︎

  2. Precision at any threshold is TP/(TP+FP). With fewer actual positives in the holdout, each flagged batch contains fewer true hits, so precision is lower at every recall level and the PR curve shifts down even for the same ranking. The baseline connection makes this concrete: the random-classifier baseline sits exactly at the positive rate, as the dotted line showed. If your holdout has 3% defaults instead of 6.7%, that floor drops too, and a model with identical ranking quality scores lower AP by construction.↩︎

  3. Pair-counting check on the same example: of the 6 possible (defaulter, non-defaulter) pairs, 5 have the defaulter scoring higher → area = 5/6 ≈ 0.833.↩︎

  4. An earlier pass through this analysis had the downsampled LGBM model on Sparkov looking like a clear winner, with early stopping running four times longer and AP moving up. That turned out to be a bug: the early-stopping validation set wasn’t guaranteed disjoint from the downsampled training set, so the model was partly being “early stopped” against rows it had already trained on. Once the split was fixed to keep that validation set clean, the apparent win vanished.↩︎

  5. My first guess was that early stopping was just being impatient, so I re-ran it with the patience raised from 50 rounds to 200 and then 500. The result doesn’t move at all, to four decimal places: iteration 7 really is the best this model gets, and letting it run to 300 trees with early stopping off drops it to AP 0.039.↩︎