How to detect model drift

Quick answer

Model drift splits into two problems. Data drift means the input distribution changed; catch it with a distribution test (PSI, KS test, or Chi-Square). Concept drift means the input-output relationship changed; catch it by tracking performance against fresh ground truth. Run both checks on a schedule, not once at launch.

1. Freeze a baseline: your training data’s feature distributions and its performance metrics.
2. Run a distribution test, PSI, KS test, or Chi-Square, on each input feature against a recent live-data window. Skipping this per-feature check is the most common mistake; an aggregate distribution can look stable while one feature has moved a lot.
3. Track model performance against ground truth separately, as labels arrive.
4. Set an explicit alert threshold (PSI above 0.25 is the standard cutoff) instead of eyeballing a chart.
5. Investigate before retraining. Drift can be a permanent shift worth retraining for, or a temporary anomaly that resolves on its own.

What is model drift?

Model drift is a model’s predictions becoming less reliable because the world it scores has changed since training. It splits into two distinct problems, and mixing them up is the most common detection mistake:

• Data drift (also called covariate shift): the statistical distribution of the input features changes, even if the relationship between inputs and outcomes hasn’t. A fraud model trained on pre-holiday spending seeing a shift in transaction amounts is a data drift example.
• Concept drift: the relationship between inputs and the target outcome changes. The same input pattern that used to mean “not fraud” now means “fraud,” because behavior changed, not the data’s shape.

A model can show zero data drift while its accuracy quietly collapses. Concept drift doesn’t require the inputs to look any different at all, which is exactly why distribution tests alone aren’t enough.

How to detect data drift

Data drift detection compares two distributions, your training baseline and a recent window of live data, without needing any ground truth labels. That makes it useful for catching problems before a performance drop is even visible.

PSI’s thresholds come from credit-scoring practice: below 0.1 is read as no meaningful change, 0.1 to 0.25 as a small-to-moderate shift worth a look, and above 0.25 as a substantial change that typically triggers action (population stability testing review, arXiv). Run the test per feature, not just on the dataset as a whole.

How to detect concept drift

Concept drift can happen without any visible change in the input data, so distribution tests alone won’t catch it. Instead, monitor how well the model is actually performing:

• Track accuracy, error rate, or a relevant metric (F1, AUC) on a rolling window of newly labeled data, compared against the training-time baseline.
• Monitor residuals for regression models. A shift in residual variance or a systematic bias appearing over time signals that the input-output relationship has moved.
• Compare recent-window performance to older windows on a fixed cadence, weekly or monthly depending on how fast the domain changes, rather than waiting for someone to notice.

The catch is ground truth. Concept drift detection needs labeled outcomes, and those often arrive on a delay (waiting to confirm whether a transaction was actually fraudulent, for instance). Build the monitoring cadence around how long that delay realistically runs, not around an arbitrary schedule.

Other detection methods worth knowing

Beyond PSI, KS, and Chi-Square, a few techniques show up in more advanced monitoring setups:

• Wasserstein distance and KL/Jensen-Shannon divergence: alternative distribution-distance measures, useful when PSI’s binning approach is too coarse.
• ADWIN: an adaptive windowing algorithm that expands its comparison window while data looks stable and shrinks it the moment two sub-windows’ means diverge, with built-in bounds on false positive and false negative rates (Bifet & Gavaldà, SIAM 2007).
• Page-Hinkley test: a sequential test that flags a change the moment a running observation drifts too far from the historical mean, originally developed for industrial quality control (Page, Biometrika 1954).
• CUSUM and Hidden Markov Models: sequential-analysis approaches for spotting a gradual shift in residuals or state, rather than a single snapshot comparison.

These matter most once real-time pipelines or adaptive windowing are in play. For most teams still building out their MLOps practice, PSI plus a performance-monitoring dashboard covers the majority of cases.

A simple monitoring workflow

• [ ] Freeze a baseline snapshot of training-time feature distributions and performance.
• [ ] Score PSI (or KS/Chi-Square) per feature on a recurring schedule against live data.
• [ ] Log rolling performance metrics against ground truth as labels arrive.
• [ ] Set explicit alert thresholds for both distribution drift and performance decay. Don’t rely on eyeballing a dashboard.
• [ ] Investigate root cause before retraining automatically. Drift can reflect a genuine, permanent shift (retrain) or a temporary anomaly (don’t retrain on noise).

FAQ

What’s the difference between data drift and concept drift?
Data drift is a change in the input feature distribution. Concept drift is a change in the relationship between inputs and the correct output. A model can have one without the other.

What PSI value indicates model drift?
Below 0.1 is generally read as stable, 0.1 to 0.25 as a small-to-moderate shift worth investigating, and above 0.25 as a substantial change that typically warrants action.

Can you detect concept drift without labeled data?
Not directly. Concept drift detection relies on comparing predictions to actual outcomes. Without ground truth, it can only be inferred indirectly, through proxy signals like a sudden change in the prediction distribution itself.

Next step

Pick one model that’s already live and run a PSI check on its top five features against last month’s data. If it’s already run into the classic model deployment challenges around silent performance decay, that single pass usually surfaces whether monitoring is worth building out further, before investing in a full pipeline.