Label Drift

Label drift is a type of data drift where the mix of outcomes a model predicts changes between training and production, even though the link between inputs and each outcome stays the same.

What It Is

A fraud model trained last year still flags transactions today, but the share of fraud it now sees has crept up, and its judgments quietly lose their edge. Label drift is one of the reasons a working model decays without throwing a single error. It happens when the frequency of the outcomes a model predicts changes in production, while the fingerprint of each outcome stays the same. The model runs fine and sounds sure of itself, so the problem hides until a metric you care about, like precision or alert volume, drifts off target.

Every classifier learns not just how to tell outcomes apart, but how common each outcome was in its training data. That background frequency is the base rate, the prior probability of each class. Label drift, also called prior probability shift or target drift, means those base rates have moved. Picture a churn model built when roughly one in twenty customers left each month. After a price change, that rate doubles. The traits of a churning customer have not changed, but there are far more of them, and a model tuned to the old rate now under-counts churn and misplaces its threshold.

What pins label drift down is which part stays fixed. The distribution of the target changes while the relationship from each label back to its features holds steady, written as P(y) changing while P(x given y) does not. That is the mirror image of covariate shift, where the inputs move and the outcome rule stays put, and different again from concept drift, where the rule connecting inputs to outcomes itself rewrites. All three sit under data drift, the umbrella term for any training-to-production gap. Because each class still looks the same, label drift is often the cheapest type to correct: you re-estimate the new base rates and recalibrate instead of rebuilding the model.

How It’s Used in Practice

Product teams most often meet label drift in production model monitoring, with one practical wrinkle: the true labels usually arrive late. Whether a customer actually churned can take weeks to confirm, so teams watch the distribution of the model’s own predictions as an early stand-in. When the predicted positive rate climbs or falls out of its normal band, that is the first sign the real outcome mix may be shifting.

Confirming the shift is a statistics problem. Teams compare the label or prediction distribution over time using tests like Population Stability Index for a single drift score, or a chi-square test for categorical outcomes. Open-source tools such as Evidently, NannyML, Alibi Detect, and whylogs package these checks, and some estimate how performance will move before ground truth lands. When the base rate has shifted, the fix is often to recalibrate the model’s decision threshold or re-weight its class priors, and to retrain on fresh data only when recalibration is not enough.

Pro Tip: Watch your model’s output distribution, not just its accuracy, because true labels often lag by weeks. If the predicted positive rate drifts but each class still looks the same, reach for threshold recalibration before you queue an expensive retrain. It is faster, and it fixes the actual problem.

When to Use / When Not

Common Misconception

Myth: You cannot measure label drift until the true labels come back, so there is nothing to do but wait. Reality: The model’s own predicted-label distribution is a usable early proxy. If predictions skew while each class still looks the same, you can re-estimate base rates and recalibrate before ground truth confirms it, often catching the decay weeks earlier.

One Sentence to Remember

Label drift is the model going stale because the world changed how often each outcome happens, not what each outcome looks like. Track the distribution of your predictions, not only your error rate, and reach for recalibration before retraining.

FAQ

Q: What is the difference between label drift and covariate shift? A: Label drift changes how often each outcome occurs, the distribution of the target. Covariate shift changes the distribution of the input features instead. Both are forms of data drift, but they move different parts of the data.

Q: How do you detect label drift before the true labels arrive? A: Monitor the distribution of the model’s predictions as a proxy for the real outcome mix. Tests like Population Stability Index or a chi-square test flag when the predicted-label distribution moves outside its normal range.

Q: How do you fix label drift? A: Often you can recalibrate the decision threshold or re-weight the class priors to match the new base rates, which is cheaper than retraining. Retrain on fresh data only when the classes themselves have started to look different.