Imbalanced Learning & Cost-Sensitive Evaluation

In fraud, default and AML problems the positive class is rare, often well under 1%. Models trained naively learn to predict "not fraud" and look great on accuracy. This lesson covers the techniques that actually work.

Pick the right metrics first

• PR-AUC (average precision): focuses on the rare class. Prefer it to ROC-AUC under heavy imbalance.
• Recall at a fixed precision (or at a fixed alert rate): matches how operations teams work.
• Cost-weighted metrics: expected loss = FN × fraud cost + FP × review/friction cost.
• Calibration: if a probability feeds a pricing or limit decision, check reliability curves or the Brier score.

Data-level techniques

Algorithm-level techniques

• Class weights: class_weight="balanced" (scikit-learn) or scale_pos_weight ≈ negatives / positives (XGBoost, LightGBM).
• Focal loss: down-weights easy negatives in neural networks.
• Anomaly detection: Isolation Forest or autoencoders when labels are very scarce. Use them as features or as a first filter.
• Ensembles of balanced subsets (EasyEnsemble, BalancedRandomForest).

Undersampling changes probabilities: correct them

If you keep only a fraction β of the negatives, the model's predicted probability p_s is inflated. The corrected probability is:

[ p = \frac{\beta\, p_s}{\beta\, p_s - p_s + 1} ]

Or simply recalibrate on untouched validation data with Platt scaling or isotonic regression.

Choose the threshold by cost

import numpy as np
def best_threshold(y, p, cost_fn=5000, cost_fp=50):
    ts = np.linspace(0.01, 0.99, 99)
    costs = [((p < t) & (y == 1)).sum() * cost_fn + ((p >= t) & (y == 0)).sum() * cost_fp for t in ts]
    return ts[int(np.argmin(costs))]

• Add operational constraints, for example "at most 2,000 alerts a day", and pick the best threshold that satisfies them.

Validation design

• Use out-of-time validation (train Jan–Sep, validate Oct–Dec).
• Use grouped splits by customer, so the same customer never appears in both train and test.
• Report the metrics with confidence intervals; rare positives make estimates noisy.

Check yourself