"Don't Break It" and "Ship Faster": How SRE Settles the Conflict
Agree on an SLO and treat the gap below 100 percent as an error budget. While budget remains, the team ships at full speed. When burn is fast or the budget is gone, a pre-agreed policy slows or gates releases for that service until reliability recovers. The SLO, not seniority, decides.
Two instructions that sound like a contradiction
"Don't let it go down" and "ship faster" usually arrive from different people on the same day. Product wants the feature out this sprint. The VP who took the escalation from the last outage wants zero incidents. Both are reasonable on their own. Together they are an unbounded demand, because every deploy is a change to a running system and change is where much of the risk lives.
Without a shared rule, the conflict gets settled by whoever shouted last. After a bad week the team freezes releases informally. After a quiet month product pushes a risky migration through on a Friday. Neither decision is based on how reliable the service actually is. Site reliability engineering's answer is to stop arguing about adjectives and agree on a number.
The error budget turns the argument into arithmetic
Start with a service level objective measured from the user's side, such as the proportion of checkout requests that succeed within a latency threshold. Set a target below 100 percent, because 100 percent is neither achievable nor something users can perceive. The gap between the target and perfection is the error budget.
A 99.9 percent availability SLO over 30 days allows 0.1 percent failure. For a request-based SLI that is one failed request in a thousand. Expressed as time, it is about 43 minutes of total unavailability across the window (30 days times 1,440 minutes times 0.001). That budget is not a target to hit. It is the amount of unreliability the business has explicitly agreed users will tolerate.
Once the budget exists, both demands become claims on the same resource. Shipping spends budget, because some releases fail. Hardening work saves it. The question changes from "is this safe?" to "can we afford this risk right now, given what we have already spent?"
The policy is the negotiation, done in advance
An error budget without a policy is a dashboard. The value comes from agreeing, before anything breaks, what the team does at each level of spend. A workable policy answers four questions.
What happens while budget remains
Normal release cadence. The team ships at whatever pace its delivery pipeline supports. Nobody needs reliability sign-off for routine changes.
What happens when burn is fast
If the budget is being consumed much faster than the window allows, releases slow down and the on-call engineer gets authority to pause deploys until the cause is understood. This is a speed limit, not a freeze.
What happens when the budget is gone
Feature releases stop for the affected service. Engineering time moves to the reliability work the postmortems identified. Security fixes and changes that reduce risk still ship.
Who can override it
Name the person, usually a senior product or engineering leader, and require the override to be written down with the reason. Overrides will happen. Making them visible is what keeps the policy honest.
Release gating that follows the budget
Gating is where the policy becomes mechanism. The simplest version is a check in the deploy pipeline that reads remaining budget and current burn rate from the monitoring system and blocks or warns on production deploys for that service when thresholds are crossed. The gate should name the service and SLO it is protecting, so an engineer who hits it knows exactly why.
Keep the gate narrow. Gate the service whose SLO is burning, not the whole organisation. Allow labelled exceptions for rollbacks, security patches, and fixes for the incident that caused the burn, since blocking the fix makes the outage longer. Log every exception. A gate that blocks everything for everyone gets bypassed within a month.
Make each change cheaper instead of choosing a side
The deeper answer to "don't break it" and "ship faster" is that they stop conflicting when the cost of a bad change falls. DORA's research programme has generally found that delivery throughput and stability tend to move together in high-performing teams rather than trading off, and the mechanisms behind that are familiar.
Small batches mean each failure has a small blast radius and an obvious suspect. Canary releases expose a change to a slice of traffic and compare its SLI against the baseline before promoting it. Feature flags separate deploying code from exposing behaviour, so a bad feature is switched off rather than rolled back. Fast, tested rollback turns a bad deploy into minutes of budget instead of hours. Each of these lets the team ship more often while spending less budget per release.
Reading the budget in both directions
Teams treat a depleted budget as a signal and ignore the other one. A budget that is never spent, quarter after quarter, says the team is being more cautious than users need. That caution has a cost in delayed features and slower learning. It is a legitimate argument for shipping faster, migrating sooner, or running that risky experiment, and SREs should be the ones making it.
This is what makes the error budget a negotiation tool rather than a weapon for one side. Product gets a principled case for speed when reliability is healthy. Engineering gets a principled case for hardening when it is not. Neither depends on who has more organisational weight.
Where this goes wrong
An SLO set tighter than users need burns constantly and turns the policy into a permanent freeze, which leadership will rightly override. An SLI that measures server health instead of user experience lets the budget look healthy while customers suffer. A policy written by the SRE team alone, without product agreeing to it, is ignored at the first deadline. And a budget that resets every month can encourage end-of-window risk taking; rolling windows reduce that.
The fix in each case is the same: revisit the SLO and policy on a schedule, with product and engineering both in the room, using the incident history as evidence.
Frequently asked questions
- What is an error budget in SRE?
- An error budget is the amount of unreliability a service may have over a window while still meeting its SLO. A 99.9 percent target allows 0.1 percent of requests to fail. The team spends it on releases and experiments, and when it runs out, feature work pauses in favour of reliability work.
- Should releases stop completely when the error budget is exhausted?
- Feature releases for the affected service usually stop, but not every change. Rollbacks, security patches, and fixes for the cause of the burn should still ship, because blocking them makes reliability worse. The policy should list these exceptions in advance and require each one to be logged.
- How do you get product managers to accept an error budget policy?
- Show them it works both ways. An unspent budget is a documented argument for shipping faster, not just a reason to slow down. Agree the SLO target and the policy together, name who can override it, and review both against real incident history every quarter.
- Is reliability always a trade-off against deployment speed?
- Not at the margin teams usually fight over. Small batches, canary releases, feature flags, and fast rollback lower the cost of each failed change, so a team can deploy more often while spending less error budget per release. The trade-off is real only when each change is large and hard to undo.