How to detect AI model drift before your customers do

Every deployed model quietly degrades. The world moves, inputs shift, user behaviour changes, and the model that was 94% accurate at launch is 82% accurate six months later — and nobody noticed until a customer complained. Drift detection is the discipline of catching that decay before it hits users. This post covers the three kinds of drift, the signals to watch, and how to wire the whole thing into a governance workflow.

The three kinds of drift

People say "drift" like it is one thing. It is three, and the mitigation is different for each.

Concept drift

The relationship between inputs and the target changes. The features look the same, but the correct answer for those features is different. A fraud model trained before a new attack pattern emerged will keep classifying transactions confidently while the ground truth has shifted underneath it.

Data drift (covariate shift)

The input distribution changes even though the underlying relationship is stable. A recommendation model built on last quarter's traffic mix will misbehave when a new user segment shows up with different behaviour. The model has not become wrong, it has become out-of-domain.

Label drift (prior probability shift)

The distribution of the target changes. The base rate of the positive class goes up or down for reasons unrelated to the features. A churn model trained during a stable pricing period will misfire once you launch a new plan that doubles the churn rate.

Signals to watch

You do not need exotic statistics. Three or four well-chosen signals, computed daily, will catch most real drift.

• →Population Stability Index (PSI) on each input feature — compares the current distribution to the training distribution, fires when a feature has shifted more than a defined threshold
• →Kolmogorov-Smirnov (KS) statistic on continuous inputs and outputs — simple, non-parametric, works with modest sample sizes
• →Output distribution drift — the distribution of predicted scores or classes over time, independent of any ground truth
• →Rolling accuracy or calibration error where labels arrive within a reasonable delay — the ground truth signal, when you can get it

Where to put the check

Two placements, chosen based on latency budget and traffic volume.

• →Post-inference lightweight check — after each prediction, log the input features, output, and any model confidence. Cheap and captures everything.
• →Batch monitoring job — a scheduled job (hourly or daily) reads the inference log, computes PSI, KS, and output drift, and writes results to a metrics store

Do not try to compute drift statistics inline on every request. It adds latency, and the numbers are noisy on small windows anyway. Batch is the right shape.

What to log

The value of drift detection lives or dies on log quality. At minimum:

• →Model version — from your model registry, immutable per training run
• →Input feature vector (or a hash if the raw values are sensitive)
• →Output — predicted class or score, plus confidence if available
• →Timestamp with timezone
• →Request context — feature or endpoint that invoked the model, enough to slice drift by use case
• →Ground truth reference — a foreign key you can use to join labels back later when they arrive

When to retrain

Retraining is not automatic. Drift is a signal for a human decision. A useful rule of thumb: retrain when two or more of your signals cross their thresholds simultaneously and the crossing persists for more than one review window. Single-day spikes are usually noise or seasonality.

Before retraining, ask whether the drift is caused by a data pipeline change, a business change, or a genuine model-decay event. Retraining on drifted data that reflects a broken upstream feed will bake the bug into the next version.

Wire the signal into governance

Drift is a risk-register entry, not an ops-only concern. When a drift alert fires beyond a defined tolerance, that event should create or update a risk record with the model version, the affected feature, the observed magnitude, and the responsible owner. Auditors do not want to see that you have monitoring — they want to see that monitoring produced decisions and evidence.

How Zilonex Govern helps

Zilonex Govern ties drift monitoring directly to the AI model registry and risk register. When your batch monitoring job flags a threshold breach, Govern opens a risk item against the correct model version, assigns the recorded owner, and captures the drift metrics as audit evidence. You get drift monitoring that is not just an ops dashboard, but a governance artefact.