Designing for LLM Provider Outages

Hosted model providers publish incidents on public status pages, and those incidents are frequently partial: elevated errors on one model, degraded latency, failures in a specific feature such as file handling or tool use, problems with authentication, or issues limited to a region or a product surface. Coding assistants, chat products, and APIs from the same provider can fail independently.

For a product built on the API, partial matters. A total outage is obvious. A 5 percent error rate on one model, or latency doubling, quietly degrades your product and may not show on your dashboards unless you measure it.

Status pages are updated by people, after investigation, and describe the provider's view. Your users experience your view.

• Instrument every model call with model, operation, status, latency, and token counts.
• Alert on symptoms: error rate and latency percentiles per model, and the proportion of requests falling back to degraded paths.
• Run synthetic checks with representative prompts, including tool use and structured output if you depend on them.
• Watch output quality signals where you can, such as schema validation failures, since some incidents show up as malformed output rather than errors.

Model calls are slow and variable, which tempts teams to set long timeouts and generous retries. During an outage that combination holds connections open, piles up queues, and multiplies traffic to a struggling provider.

Set timeouts per use case based on what users will tolerate. Retry transient failures with exponential backoff and jitter, cap retries per request, and use a retry budget so retries cannot exceed a fraction of normal traffic. Respect rate limit responses and any retry-after guidance. Add a circuit breaker so that sustained failures stop calls quickly and route to a degraded path.

Decide in advance what your product does without the model. The answer differs by feature:

• Assistive features can hide or show a clear unavailable state while the rest of the product works.
• Batch work can queue and resume later.
• Retrieval-backed answers can return the retrieved sources without a generated summary.
• Cached responses can serve repeated questions.
• Critical paths that cannot degrade are candidates for failover.

A product that stays usable without its model feature suffers far less from an outage than one where every page waits on a generation.

Routing to a secondary model or provider can keep critical paths running, but models are not interchangeable. Prompts, tool calling formats, output styles, context limits, and safety behaviour differ. A failover path that has never been evaluated may produce different or wrong results under pressure.

Keep failover for paths that need it, maintain prompts and evaluations for the secondary model, test failover regularly, and monitor quality when it is active. Also consider data handling: sending data to a different provider may carry different contractual or regulatory implications.

Not every provider problem appears as an incident. Hitting your own rate limits or quota during a traffic spike produces the same user-facing failures as a provider outage, and it is entirely on your side. Track usage against limits, alert well before you reach them, and know how to request increases in advance. Queue non-interactive work so it absorbs spikes instead of competing with interactive requests for the same quota.

When the fault is at the provider, the incident is still yours. Declare it, assign a lead, track which features are affected, switch to degraded modes or failover deliberately, tell users what is happening, and record impact. Close it when your metrics recover, not when the provider's status page turns green. Afterwards, review whether detection, degradation, and communication worked, which are the parts you control.