How to Handle Class Imbalance in scikit-learn: Class Weighting, Threshold Moving, and SMOTE in 2026

Your fraud model hit 99.2% accuracy in the notebook. You shipped it Friday. By Monday it had waved through every fraudulent transaction in the queue — because predicting “legitimate” every single time is how you score 99.2% when fraud is one in two hundred. The model didn’t fail. Your metric did, and nobody specified a better one.

Before You Start

You’ll need:

• A Scikit Learn classifier you can retrain
• A working grasp of Class Imbalance — why a model drifts toward the majority class when the labels aren’t even
• Your imbalance ratio as an actual number — how many majority rows you have per minority row
• For resampling, the Imbalanced Learn package, which is a separate install from scikit-learn

This guide teaches you: how to treat imbalance as a stack of three intervention points — data, algorithm, decision threshold — and pick the lightest one that hits your metric.

The 99% Accuracy Trap

Most imbalance disasters start the same way: train a model, read the accuracy score, ship. On a dataset that’s 200 majority rows to every minority row, a model that predicts the majority class for everything scores around 99.5% accuracy and catches nothing. The score isn’t wrong. It’s just answering a question you didn’t mean to ask.

It worked on Friday. On Monday, minority-class recall was zero — because nothing in the training run told the model that missing a positive case costs more than missing a negative one.

Step 1: Map the Three Layers Where Imbalance Lives

Before you touch a single parameter, locate where you can intervene. Imbalance shows up at three layers, and each technique acts on exactly one layer. Mix them up and you’ll stack fixes that quietly fight each other.

Your system has three intervention layers:

• The data layer — what the model trains on. Resampling lives here: Oversampling the minority class (including SMOTE) or Undersampling the majority. You change the training distribution before the model sees it.
• The algorithm layer — how the model weighs its mistakes. Class Weighting lives here: you tell the loss function a minority error is more expensive, a form of Cost Sensitive Learning. The data is untouched; only the penalty changes.
• The decision layer — where probabilities become labels. Classification Threshold moving lives here. The trained model is frozen; you just stop using 0.5 as the cut-off.

The Architect’s Rule: If you can’t name which layer a fix touches, you can’t predict what it breaks.

Evaluation isn’t a fourth fix, but it’s the fourth thing you map — because if you measure with accuracy, everything above it is invisible.

Knowing the layers tells you what’s possible. The next job is telling your implementation exactly which one to use, and that starts with constraints, not code.

Step 2: Specify the Constraints Before Any Resampling

An AI coding assistant — or a teammate, or you in three weeks — will guess every value you don’t pin down. On imbalance, the wrong guess is silent: the code runs, the numbers look plausible, and the minority class quietly evaporates. Specify these before anything generates a line of code.

Context checklist:

• Imbalance ratio stated — for example, 200:1. This decides whether weighting alone is enough or resampling earns its keep.
• The metric you optimize — minority-class recall, F1, or PR-AUC. Never accuracy. (Step 4 covers why.)
• Library versions pinned — scikit-learn 1.9.0 shipped June 2, 2026 with Python 3.11–3.14 support (scikit-learn release notes), and imbalanced-learn 0.14.2 followed on June 7, 2026 (imbalanced-learn changelog). Pin both; imbalanced-learn tracks scikit-learn’s API closely, so a mismatch tends to break at import.
• Where resampling runs — training folds only, never the full dataset. This single rule is what prevents Data Leakage.
• Categorical features flagged — plain SMOTE interpolates between rows and invents impossible values for categorical columns. If you have them, the spec has to say so.

The Spec Test: If your context doesn’t say “resample inside the Cross Validation pipeline,” the AI will resample the whole dataset first — and your validation scores will lie to you, in your favor.

One boundary trips up half the imbalance tutorials online: SMOTE and ADASYN are not part of scikit-learn. They live in imbalanced-learn — you install imbalanced-learn, you import imblearn. Treat it as a separate dependency with its own version.

Step 3: Sequence the Fix from Cheapest to Most Invasive

Order matters because each layer adds cost and risk. Start with the intervention that changes the least, measure, and escalate only when the metric tells you to. That’s the opposite of reaching for SMOTE first because it’s the famous one.

Build order:

1. Class weighting first — set class_weight='balanced' on the estimator. scikit-learn computes the weights as n_samples / (n_classes * np.bincount(y)), so rarer classes get proportionally heavier penalties; it’s available on LogisticRegression, RandomForestClassifier, SVC, and more (scikit-learn Docs). One parameter, no new data, no leakage surface. This is your baseline.
2. Threshold moving second — if recall is still short, stop hand-rolling threshold loops. Wrap the fitted model in TunedThresholdClassifierCV, added in scikit-learn 1.5 (scikit-learn Docs), which searches the cut-off on predict_proba or decision_function for you. By default it optimizes balanced accuracy over 5-fold stratified cross-validation (scikit-learn Docs) — change the scoring to match your real objective.
3. Resampling last — only when weighting and thresholding still leave too little minority signal to learn from. Reach for SMOTE, or ADASYN when you want synthetic counts to adapt to local density instead of filling uniformly. ADASYN imports from imblearn.over_sampling with sampling_strategy='auto' and n_neighbors=5 by default, concentrating samples in the sparse, hard-to-learn regions (imbalanced-learn Docs).

Pin one rule above all else: resampling goes inside an imblearn Pipeline so it fits on training folds only during cross-validation, never on the full dataset before the split (imbalanced-learn Docs). That one structural choice is the line between honest scores and self-deception.

For each technique, specify what it receives (the training fold), what it returns (a rebalanced fold or a recalibrated threshold), and what it must NOT touch (the test fold). If you move to neural networks later, Focal Loss fills the algorithm-layer role weighting plays here — but it lives in deep-learning frameworks, not scikit-learn.

Compatibility notes (imbalanced-learn 0.14.2):

• n_jobs is deprecated on SMOTE, ADASYN, BorderlineSMOTE, SMOTENC, SMOTEN, and SVMSMOTE. Pass a nearest-neighbors estimator with n_jobs set instead of the sampler’s own parameter (imbalanced-learn changelog).
• Pin both libraries. imbalanced-learn 0.14.2 post-dates scikit-learn 1.9.0, but the tested upper bound is not stated on the install page — version-pin both rather than asserting a compatibility ceiling.

Step 4: Validate Against the Minority Class, Not the Average

Validation is where imbalance projects live or die, because the default metric is built to hide your failure. Accuracy averages over every row, so on a 200:1 split the majority drowns out the signal you care about. Swap it for metrics that look at the minority class directly.

Validation checklist:

• Read the confusion matrix first — failure looks like: a fat column of false negatives, nearly every positive case predicted negative.
• Report Balanced Accuracy, not raw accuracy — failure looks like: 99% accuracy sitting next to roughly 50% balanced accuracy; that gap is the imbalance lying to you.
• Use PR-AUC via average_precision_score — failure looks like: a flattering ROC-AUC hiding poor precision, because precision-recall curves expose minority performance that ROC smooths over (scikit-learn Docs). Prefer average_precision_score over a trapezoidal auc() of the curve; they differ, and average precision is the honest one for imbalanced data.
• Confirm resampling never saw the test fold — failure looks like: validation scores that beat production by a wide margin, the classic data leakage signature. Cross-validation with the sampler inside the pipeline is the fix.
• Track minority-class F1 — failure looks like: precision and recall that read fine alone but whose harmonic mean has cratered.

Common Pitfalls

Pro Tip

Write down your imbalance ratio and your target metric before you pick a single technique — in a comment, a docstring, anywhere your AI assistant will read it. Every imbalance decision flows from those two numbers. Specify them once and technique selection stops being a guess and starts being arithmetic. Skip them and every future session re-litigates the same wrong default.

Frequently Asked Questions

Q: How to handle class imbalance in scikit-learn step by step? A: Set class_weight='balanced', measure minority recall, then tune the threshold with TunedThresholdClassifierCV. Resample with SMOTE only if both fall short. One detail the steps gloss over: for tree ensembles, weighting alone usually suffices — SMOTE pays off most for linear and distance-based models.

Q: When should you use class weighting instead of resampling? A: Use class weighting as the default — it’s one parameter, adds no data, and creates no leakage surface. Switch to resampling when the minority class is too small for the model to learn its shape from weighting alone, or when your estimator ignores class_weight. Weighting changes penalties; resampling changes the data.

Q: How to use PR-AUC and F1 score to evaluate an imbalanced classifier? A: Compute PR-AUC with average_precision_score and F1 with f1_score, both focused on the minority class. PR-AUC summarizes the precision-recall trade-off across all thresholds; F1 scores one chosen threshold. Report both: strong PR-AUC with weak F1 means your threshold is wrong, not your model.

Your Spec Artifact

By the end of this guide, you should have:

• A three-layer map of your imbalance problem — which technique acts on the data, the algorithm, and the decision threshold
• A constraint list — imbalance ratio, target metric, pinned library versions, the leakage rule, categorical flags
• A validation checklist — PR-AUC, minority-class F1, balanced accuracy, the confusion matrix, and a leakage check

Your Implementation Prompt

Drop this into Claude Code, Cursor, or Codex when you’re ready to build. Fill every bracket with your own values — they map one-to-one to the Step 2 constraint checklist.

You are implementing class imbalance handling in scikit-learn. Follow this spec exactly.

CONTEXT (fill these in):
- Imbalance ratio: [e.g. 200:1 majority:minority]
- Target metric: [minority-class recall | F1 | PR-AUC]
- Library versions: scikit-learn [1.9.0], imbalanced-learn [0.14.2]
- Categorical features present: [yes/no -- list the columns if yes]
- Estimator: [e.g. LogisticRegression | RandomForestClassifier]

BUILD ORDER (do not skip ahead):
1. Split train/test FIRST, stratified on the target. Never resample before this split.
2. Baseline: fit [estimator] with class_weight='balanced'. Report minority-class recall and PR-AUC.
3. If recall < [target], wrap the model in TunedThresholdClassifierCV with scoring=[your metric]. Re-report.
4. If still short, add SMOTE (or SMOTENC if categorical features = yes) INSIDE an imblearn Pipeline so it fits on training folds only.

VALIDATE:
- Print the confusion matrix and balanced_accuracy_score.
- Report PR-AUC via average_precision_score on the minority class.
- Confirm the sampler sits inside the cross-validation pipeline, not applied to the full dataset.

CONSTRAINTS:
- Do not optimize for or report plain accuracy.
- Do not pass n_jobs to the sampler (deprecated); set it on the nearest-neighbors estimator instead.
- Pin both library versions in requirements.

Ship It

You now read imbalance as three layers, not one panic. You know which technique touches which layer, the order to try them in, and the metrics that won’t lie to you. The next imbalanced dataset isn’t a problem to fear — it’s a spec to fill in.