Should every incident have a root cause?

Every incident should have an explanation, but rarely a single cause. Most serious incidents combine a trigger with contributing factors such as missing safeguards, risky rollout practices, or detection gaps. Recording only one cause hides the systemic factors that are usually the most valuable to fix.

What are good root cause categories for incidents?

Use layers rather than one list. Triggers such as deploys, configuration changes, dependency failures, and expiries; contributing factors such as missing validation, unsafe rollout, retry behaviour, and single points of failure; and detection gaps such as missing or noisy alerts. Keep each list around ten items, define each clearly, and allow unknown.

Why not use human error as a root cause?

Because it explains nothing actionable. People make mistakes in every system; the useful questions are why the system made this mistake easy and why its impact was large. Coding human error ends the investigation early, discourages honest reporting, and hides the design and tooling problems that actually need fixing.

Designing Incident Cause Codes When Outages Have Many Causes

Record causes in layers instead of one field: the trigger that started the incident, the contributing factors that let it spread, and the detection or response gaps that made it last. Keep each list short, allow several contributing factors per incident, allow an explicit unknown, and never offer human error as a category. Review the codes quarterly against real postmortems.

Why one root cause is the wrong model

Look at any serious postmortem and the story has several threads. A configuration change triggered the fault. A missing validation let the bad value through. A retry policy turned a partial failure into a full one. The alert that should have caught it was silenced months ago. Remove any one thread and the incident is smaller or never happens.

A dropdown asking for the root cause forces someone to pick one thread. They usually pick the trigger, because it is closest in time and easiest to name, and the systemic factors disappear from the data. Over a year, the reports say most incidents are caused by changes, which is true and useless.

Why you still want codes

The narrative belongs in the postmortem. Codes serve a different purpose: seeing patterns across dozens or hundreds of incidents that no single review reveals. Which contributing factors recur? Which services keep appearing? Are detection gaps shrinking? That requires structured, consistent data. The goal is codes that support that analysis without pretending any one incident is simple.

A layered structure

Capture three layers, each with its own short list.

Trigger (one)

What started the incident. Examples: code deploy, configuration change, infrastructure change, dependency failure, traffic change, certificate or credential expiry, hardware or capacity failure, security event, unknown.

Contributing factors (several)

What allowed the trigger to become an incident, or made it worse. Examples: missing or insufficient validation, missing test coverage, unsafe rollout without staged deployment, retry or timeout behaviour, capacity headroom, single point of failure, stale or missing runbook, unclear ownership, configuration drift.

Detection and response gaps (several, optional)

What made it last longer than it should. Examples: no alert, alert too noisy to act on, alert routed to the wrong team, slow escalation, missing access for responders, unclear rollback path.

Rules that keep the data honest

Keep lists short. Around ten items per layer. Long taxonomies get used inconsistently, and most incidents end up in whichever option is nearest the top.
Define every code in one sentence, with an example. Two engineers should code the same incident the same way.
Allow unknown. A forced guess is worse than an honest gap. A rising share of unknowns is itself a signal about investigation quality.
No human error. A person making a mistake is a starting point for asking why the system made the mistake easy and its effect large. Code the system factor instead: missing validation, unsafe tooling, unclear runbook.
Code after the review, not at close of the ticket. First-hour guesses are often wrong. If the ticket must close before the postmortem, allow the codes to be revised.
Review the list quarterly. Merge codes nobody uses, split codes that hide two different things, and check a sample of incidents for consistent coding.

A worked example

A configuration push sets a connection pool limit too low on one service. Requests queue, clients time out and retry, and the retries overload a shared database, which takes down three other services. The alert on the first service had been downgraded to a warning after too many false positives, so the page came from customer reports forty minutes in.

With a single field, this is coded as configuration change. With layers, it is trigger: configuration change; contributing factors: missing validation on the setting, retry behaviour without backoff, shared database as a single point of failure; detection gap: alert too noisy to act on.

The second version tells you four things to fix. The first tells you not to change configuration, which nobody will follow.

Using the data

Report on contributing factors, not triggers. Knowing that changes trigger most incidents tells you nothing new. Knowing that unsafe rollout without staged deployment contributes to a large share of change-triggered incidents tells you exactly what to fix.

Track detection gaps separately. Time to detect is often the largest share of impact, and it improves only if someone watches it.

Many ticketing tools, SubZero included, can require a cause code before an incident closes. That is useful for consistency, as long as the taxonomy behind it follows the rules above.

Frequently asked questions

Should every incident have a root cause?: Every incident should have an explanation, but rarely a single cause. Most serious incidents combine a trigger with contributing factors such as missing safeguards, risky rollout practices, or detection gaps. Recording only one cause hides the systemic factors that are usually the most valuable to fix.
What are good root cause categories for incidents?: Use layers rather than one list. Triggers such as deploys, configuration changes, dependency failures, and expiries; contributing factors such as missing validation, unsafe rollout, retry behaviour, and single points of failure; and detection gaps such as missing or noisy alerts. Keep each list around ten items, define each clearly, and allow unknown.
Why not use human error as a root cause?: Because it explains nothing actionable. People make mistakes in every system; the useful questions are why the system made this mistake easy and why its impact was large. Coding human error ends the investigation early, discourages honest reporting, and hides the design and tooling problems that actually need fixing.