How to Investigate Incidents With an SRE Copilot Without Trusting It Blindly
Give the copilot read-only access to telemetry, deploy history, and past incidents, then delegate evidence gathering: recent changes, breakdowns by version and region, log clustering, and similar incidents. Require the query behind every claim, verify time range, scope, and field names yourself, and keep hypothesis choice, mitigation, and customer updates with responders.
Treat the copilot as a fast, unaccountable investigator
An SRE copilot here means a language model with tool access to the places incident evidence lives: metrics, logs, traces, deploy history, the incident channel, and past tickets. It can run queries, read results, and summarise faster than a person can type. It cannot be paged at 3am next month to explain why it was wrong.
That asymmetry sets the working model. The copilot gathers and proposes. The incident commander and the responders decide what is true and what to do. Every statement it makes is a lead until someone has checked the evidence behind it. Teams that hold this line get real speed. Teams that let the model's confident summary become the incident narrative eventually chase the wrong cause for an hour.
Prepare before the first incident
Most of the value is decided by setup, not by prompting under pressure.
Scope access to read-only
Investigation needs read access to telemetry, deploy logs, configuration history, and tickets. It does not need write access to production. Give remediation permissions, if ever, through a separate, audited path with human approval.
Feed it the context your humans use
Service ownership, dependency maps, runbooks, and SLO definitions are what turn a generic model into a useful one. A copilot that does not know which services sit behind the checkout path will guess, and its guesses will sound reasonable.
Decide where its output goes
Pick one place, usually the incident channel or timeline, where copilot findings are posted and labelled as machine-generated. Scattered private chats with a model leave no trail and duplicate effort across responders.
What to delegate during the investigation
Delegate the work that is mechanical, broad, and easy to check.
Change discovery is the best first ask: list deploys, feature flag flips, config pushes, and infrastructure changes for the affected services in the window before symptoms began. Then breadth queries a human would take minutes to write: error rate by version, by region, by customer tier, by endpoint. Log clustering, which groups thousands of error lines into a handful of distinct signatures, is another good fit. So is searching past incidents and postmortems for similar symptoms, and drafting queries in a query language the responder knows less well.
Keep hypothesis selection, severity, mitigation choice, and anything customer-facing with people. The copilot can rank hypotheses. A responder picks which one to test, because that choice depends on blast radius and reversibility, which the model rarely understands fully.
Verify every claim against its evidence
The rule is simple: no claim without a query. Ask the copilot to attach the exact query, time range, and data source for anything it asserts, and make that a standing instruction rather than a per-question request. Then check the parts models get wrong most often.
Time
Confirm the time range and time zone. A query over the wrong hour produces a clean, wrong answer. Check that "the error spike started after the deploy" holds at one-minute resolution, not just on a daily chart.
Scope
Confirm which service, cluster, and environment the query hit. Staging and production often share metric names.
Existence
Confirm that metric names, log fields, and hostnames in the answer actually exist. A model can produce a plausible metric that no exporter emits, then reason confidently about its imaginary values.
Disconfirmation
Ask what evidence would rule the hypothesis out, then run that query. Models are good at building a coherent story from partial data; the corrective is to look for data that breaks the story.
Hallucination risks specific to incidents
General advice about model errors undersells how incidents amplify them. Correlation with the most recent deploy is the classic trap: something always deployed recently, and a copilot asked "what changed?" will find it and imply causation. Anchoring on a similar past incident is another, because symptoms that look alike can come from different causes. Summaries of a long incident channel can drop the one message that mattered or attribute a finding to the wrong person. And if the copilot reads from a cache or an index, its view of the system can lag reality by minutes, which matters during a fast-moving failure.
None of this is a reason to avoid the tool. It is a reason to keep a human whose job in the incident is to challenge the copilot's conclusions, in the same way a good incident commander challenges the loudest responder.
Keep an evidence trail
Post copilot findings into the incident timeline with the query and a link to the result. Mark which conclusions were verified by a person and which were not. When a responder acts on a copilot suggestion, record that too.
This pays off twice. During the incident, a new responder joining can see what has been checked and what is still assumption. After it, the postmortem can ask honest questions: did the copilot shorten diagnosis, did it send the team down a false path, and which context was it missing. Those answers tell you what to fix in its setup, which is the only way it improves on your systems.
Status updates and handoffs
Copilots draft status updates and handoff summaries well, because those are summarisation tasks over material that is already in the channel. Have a human edit and send every external update. Tone, commitments about timing, and what to disclose are judgement calls with consequences, and a draft that overstates progress costs more trust than a late update.
Frequently asked questions
- What should an SRE copilot be allowed to do during an incident?
- Read telemetry, deploy history, configuration changes, tickets, and past postmortems, run queries, and post findings to the incident timeline. Mitigation, severity decisions, and customer communication should stay with responders. If remediation is ever automated, route it through a separate, audited path with explicit human approval and narrow, reversible actions.
- How do you catch an AI copilot hallucinating during an investigation?
- Require the query, data source, and time range behind every claim, then check them. Confirm metric names and log fields actually exist, confirm the environment and time zone, and ask what evidence would disprove the hypothesis. Run that disconfirming query yourself before acting on the conclusion.
- Does using an AI copilot change how postmortems should be written?
- It adds questions. Record which findings came from the copilot, which were verified, and which actions it suggested. The postmortem should note whether it sped diagnosis or led the team astray, and what missing context, such as dependency maps or runbooks, would have improved its answers.