A phased software rollout plan, and the order that holds
Going live everywhere on Monday is not a rollout plan, it is a bet placed on a weekend. Reliability engineering has a well documented alternative, and most of it transfers to a business with four branches rather than four data centres.
The plan arrives in one sentence surprisingly often. The new system goes live on the first of the month, everywhere, and the old one is switched off the same day. It sounds decisive. What it does is schedule every unknown in the project to arrive simultaneously, on the day with the least slack in it, with no way back.
There is a better shape for this, and it is written down. Large scale operators have published how they release changes into production without taking the service down with them, and the core of it applies at any size. What follows is that method translated for a business rolling a system out across branches, teams or depots.
What a canary actually is
The term gets used loosely, so it is worth taking the precise version. Google’s Site Reliability Engineering workbook chapter on canarying releases defines it as “a partial and time-limited deployment of a change in a service and its evaluation”. The part that receives the change is the canary. Everything still on the old version is the control.
Two words carry the weight. Partial, because not everyone is exposed. And time-limited, because a phase that runs until somebody remembers to end it is a permanent fork with two versions of the truth in it.
The control matters as much as the canary, and it is the part businesses skip. If every branch moves at once there is nothing to compare against, so when order volumes dip you cannot tell whether the system or the week caused it.
Why the arithmetic favours a small first group
The chapter gives a worked example that makes the case better than any argument about caution. A canary covering 5% of traffic, failing 20% of its requests, produces an overall error rate of 1%. The failure is severe inside the canary and barely visible outside it.
The logic holds exactly for a business. One branch of twenty getting a fifth of transactions wrong is a bad morning in one shop and a support queue you can staff. The same fault everywhere is a company-wide stoppage. Exposure is the only variable you control before you know whether the thing works.
Stages rather than a switch
The recommended shape is a gradual canary with several stages. The first is small and relies on the clearest signals, the chapter naming crashes and request failures. Once it passes, the next uses a larger population, which builds the volume the analysis needs.
One more rule from the same chapter transfers directly and gets broken constantly: canary deployments should be run one at a time. Roll out to a second branch while changing the pricing rules in the first and you have two changes in flight with no way to attribute what happens next. One change, one stage, one answer.
Choosing which branch goes first
This is where rollout plans usually go wrong, in one of two directions: the quietest branch goes first because it looks safest, or the most enthusiastic manager volunteers.
Both break the chapter’s requirement that the canary be sizeable enough and run long enough to be representative of the overall deployment. The quietest branch will not generate the volume for a problem to surface, and the chapter notes that performance defects often only appear under heavy load. You get a clean result that means nothing, and that clean result is what makes the full rollout feel safe.
- Busy enough that a fault shows up within a day rather than a fortnight.
- Ordinary enough that its workflow is not a special case the other branches do not share.
- Staffed by people who will say it is broken instead of quietly reverting to paper.
- Not simultaneously absorbing another change, a refit, a stocktake or a staff shortage.
The few signals worth watching
The instinct at go-live is to watch everything, which means watching nothing. The chapter is direct: stack-rank your metrics and take the top few, perhaps no more than a dozen, choosing ones that indicate problems in the service.
It also names a trap worth borrowing: exclude 400-level response codes, because they often reflect user error rather than a fault in the service. The translation is not to ignore user confusion, which matters enormously in week one, but to count it in a separate column. Staff who cannot find the discount button are a training signal; a server returning errors is a rollback signal. Mixing them gets a rollout abandoned because it looked broken when it was merely unfamiliar.
And compare the canary against the control separately rather than looking at the aggregate. The chapter warns that service-wide numbers can hide the canary’s effect, which is arithmetic rather than opinion: one branch in twenty cannot move a company total enough to notice. Look at that branch against a comparable one still on the old system, in the same week, and the comparison becomes legible.
How long each stage runs
The chapter declines to give a universal duration, which is itself informative. Its example uses 5% of traffic for 30 minutes, it ties duration to release cadence with frequent deployments warranting shorter canaries, and it suggests tuning from the failure rates of past canaries. It gives one firm floor: for non-interactive systems the canary should at minimum span a single work unit.
That floor is the useful one, because almost everything a business cares about is a work unit with a cycle: a trading day that has to close and reconcile, a delivery round that has to come back, a payroll run, a month end. A stage ending before the first full cycle has tested the easy half of the system, and reconciliation is where the expensive faults live.
One more detail that sounds technical and is not: the chapter advises that your metric interval be the same as or shorter than the canary duration. If you only get a report at month end, a one week stage is unmeasurable. Match the stage to the rhythm of the information you actually receive.
Keeping a way back
A phased rollout is only as good as the ability to undo a phase, and that capability is a decision taken in advance rather than a reflex available on the day. The platform defaults are a useful illustration of how easily it gets lost. Kubernetes’ documentation on Deployments sets out exactly what you get unless somebody chose otherwise.
| Setting | Default | What it decides |
|---|---|---|
| strategy.type | RollingUpdate | Old instances are replaced gradually rather than all killed first, which is what Recreate does |
| maxUnavailable | 25% | How much of the service may be down mid-update, rounded down from the percentage |
| maxSurge | 25% | How much extra may run above the target count during the update, rounded up |
| minReadySeconds | 0 | A new instance counts as available the moment it reports ready, with no settling period |
| progressDeadlineSeconds | 600 | How long a stalled rollout runs before it is reported as failing to progress |
| revisionHistoryLimit | 10 | How many previous versions are retained for rollback; set to zero and a rollout cannot be undone |
Two rows matter most. minReadySeconds defaults to 0, so nothing waits to see whether a new instance stays healthy before the rollout proceeds. And the documentation states plainly that setting revisionHistoryLimit to zero means a rollout cannot be undone, because the history it would roll back to has been cleaned up.
The business equivalents are questions to settle before stage one. How long the old system stays available and who pays for it. Who may call a rollback, at what threshold, without convening a meeting. And what happens to records created during a phase that gets reverted, which is the hardest of the three and the most often unanswered.
Pace, and the rhythm inside a phase
A phased rollout is slower on paper and faster in practice. The time it appears to cost is time an all-at-once rollout spends anyway, in a worse week, with the whole business watching. The shape of that failure is in why the average is the wrong number, and the planning that precedes it in how to scope a software project.
The rhythm inside the phase matters too. A weekly session where the branch using the system shows what happened, rather than a status report describing it, is what turns a rollout into information. Set it up before stage one: weekly demos and what to ask in them.
Honest limits
There is no correct percentage, and we have not given one. The 5% above is an illustration from the source, not a recommendation; the guidance is that size and duration should make the canary representative and that you tune from your own past results. A business with three branches cannot canary at 5% of anything, and its first stage is one branch because that is the smallest unit available.
The source assumes conditions you may not have. The chapter is written for services with many instances, high request volumes and automated metric comparison. The reasoning transfers; the mechanics often do not. If your rollout unit is a shop with six staff, the evaluation is a conversation and a till reconciliation, and that is legitimate rather than second best.
The Kubernetes figures were read on 18 November 2026 and are defaults, not advice. They are here to show how much rollback behaviour is decided by settings nobody chose. Your platform differs and defaults change between versions, so check yours.
And availability is the easy failure to plan for. A staged rollout catches a system that falls over and is much weaker against one that quietly writes the wrong numbers. Data faults run for a full phase with every signal green and surface at reconciliation, which is the argument for making the first stage span a close. The recovery side is in a data backup strategy in plain terms.
Describe it. We build it.
Seven or twelve days, pay on delivery, a year of maintenance included. Bring the problem, not a spec.
Book a meetingRead next
The cost of downtime for an online business, calculated
The cost of downtime for an online business, worked out from four inputs you already have, plus what your hosting agreement has actually promised you.
ReadAn employee leave management system, and the rules it has to encode
An employee leave management system is a balance engine, not a form. The entitlement bands, the holiday calendar you cannot hardcode, and what to build first.
Read