Canary Deployment

A canary deployment routes a small percentage of live traffic to a new model version so teams can detect performance regressions and errors before committing to a full production rollout.

What It Is

When a new ML model passes offline evaluation and earns promotion in the registry, teams face a direct operational question: how do you deploy it without risking your entire production traffic if something is wrong? A canary deployment answers that question. Instead of switching all requests to the new model at once, you route a small initial slice — often around 5% — to the new version while the existing model continues handling the rest. If the new version holds up, you progressively increase the split until it takes over completely.

The name comes from the historical practice of bringing caged canaries into coal mines before miners descended. The birds were more sensitive to toxic gases than humans and would show signs of distress before conditions became dangerous for the crew. A canary deployment works on the same logic: expose a controlled fraction of real production traffic to the new model, watch for failure signals, and abort before those failures reach everyone.

In ML deployments — and particularly in a model registry workflow — canary deployment addresses a gap that offline evaluation cannot close. A model can score well on a held-out validation set and still behave unexpectedly in production. Real-world data distributions shift over time. Edge cases that rarely appear in training datasets show up regularly at scale. Preprocessing pipelines sometimes have subtle differences between training and serving environments. The canary window is your chance to find these problems before they affect all users.

The setup involves a traffic-splitting layer — a load balancer, an API gateway, or a model serving framework — configured to route requests by percentage. When a model transitions from Staging to Production in your registry (say, version 3.2 in MLflow), the deployment system begins routing a small initial share to it. You monitor key signals — latency, error rate, prediction distribution, and business KPIs — through your observability stack. If the canary version holds up over a defined observation window, you incrementally increase the share until the new version handles all traffic.

Some teams automate this progression with a canary controller: a service that reads metrics and advances or reverses the traffic split based on predefined thresholds. Others manage it manually, which is reasonable in lower-traffic environments where you need more time to accumulate meaningful data.

How It’s Used in Practice

In the context of a model registry, canary deployment typically starts right after promotion. When a model moves from Staging to Production, the deployment pipeline picks up the new version and routes an initial small percentage of traffic to it while keeping the incumbent model live for the rest.

Teams track four signals during the canary window: latency (is the new model slower?), error rate (are there unexpected prediction failures?), output distribution (are the predictions shifting in ways that affect downstream systems?), and business metrics (the key performance indicators the model is driving). The observation window varies by traffic volume — a model handling high request throughput can accumulate meaningful signal in minutes, while a low-traffic model might need days before the data is statistically reliable.

Pro Tip: Define your rollback condition before you start the canary — not after you see something suspicious. Agree on a threshold (for example, error rate above a set level or tail latency exceeding a defined ceiling) and automate the rollback trigger. Without that upfront decision, teams waste time debating whether a small anomaly is noise or a real signal while the canary is already live.

When to Use / When Not

Common Misconception

Myth: Canary deployment and A/B testing are the same thing.

Reality: Both involve traffic splitting, but the goals differ. A/B testing measures which version performs better on a business metric over a defined period — it is a measurement exercise. Canary deployment is a safety mechanism: it reduces the blast radius if the new version has a critical bug. A canary moves to 100% once it proves safe; an A/B test stops at a statistically significant winner. One is about risk reduction, the other about performance comparison.

One Sentence to Remember

A canary deployment is how you put a model from your registry into production without betting everything on the first request — you prove it works on a small slice of real traffic before switching the full load over.

FAQ

Q: What percentage of traffic should go to the canary initially? A: Start with a small fraction if your traffic volume allows meaningful signal to accumulate quickly. Very low-traffic services may need a higher initial share to collect enough requests for reliable conclusions within a reasonable observation window.

Q: How long should a canary deployment run before full rollout? A: Long enough to collect statistically meaningful data — typically hours for high-traffic services and days for low-traffic ones. Define the observation window before you start, based on the minimum sample size you need to trust the metrics.

Q: Can canary deployment work with ML models that have long inference times? A: Yes, though latency comparison becomes more critical to watch. Monitor p50, p95, and p99 latency (the 50th, 95th, and 99th percentile response times) separately — averages can hide tail latency regressions that only affect a fraction of requests but degrade user experience significantly.