Spotify's March 2022 Outage: When Service Discovery Fails

Spotify published an incident report on its engineering blog. According to that report, Spotify's backend consists of many microservices that use more than one service discovery technology, including a DNS-based system and Google Cloud Traffic Director, an xDS-based traffic control plane.

On March 8, 2022, Traffic Director experienced an outage. In combination with a bug in a gRPC client library, this caused the Spotify outage: users who were logged out of a Spotify app were unable to log back in. The report gives the timeline in UTC: reports of users being logged out began at 18:12, remediation began at 18:39, and the incident was fully mitigated at 20:35.

Spotify mitigated by rolling out configuration changes to revert affected systems to its DNS-based service discovery. Follow-up actions included working with Google Cloud to understand the failure, adding monitoring and alerting for service discovery problems, and investing in further resiliency measures. Data Center Dynamics also linked Google Cloud issues that day to problems at Discord.

In a microservice architecture, almost every request involves one service finding another. When discovery fails, healthy services cannot reach each other, and the failure spreads across unrelated features at once. Nothing is wrong with the services themselves, which can make the early investigation confusing: dashboards show healthy instances and failing requests.

The login path is especially exposed, because authentication typically touches several services in sequence. A problem that affects only some service-to-service calls can still break the one flow every user needs.

The Spotify report attributes the impact to the control plane outage in combination with a client library bug. That pairing is the important part. Well-behaved clients typically keep using the last known good endpoints when the control plane becomes unavailable, so an outage of the discovery system degrades updates rather than breaking traffic. A client that mishandles control plane errors can turn a control plane problem into a data plane outage.

For your own systems, that means testing client behaviour under control plane failure explicitly: what happens when the discovery service is unreachable, slow, or returns errors, and whether clients continue with cached endpoints.

Spotify could mitigate because it had a second, independent discovery mechanism that affected systems could switch to by configuration. Many organisations rely on a single discovery system with no alternative, which leaves only waiting for the provider to recover.

A fallback does not need to be as capable as the primary. It needs to be independent, known to work, and switchable quickly. That last point means practising the switch, not just documenting it.

Managed control planes such as Traffic Director remove a large operational burden: nobody on your team has to run and scale the discovery system. The trade-off is that its failures are outside your control and can affect many services at once.

That does not argue against using managed services. It argues for deciding, per critical path, what happens when the managed dependency fails. For many paths the answer can be to wait. For the paths every user depends on, such as authentication, an independent fallback that can be switched to by configuration is worth its cost, as this incident shows.

• Treat control planes as critical dependencies, with SLIs measured from your services' point of view.
• Test client behaviour under control plane failure, including library upgrades, which can change it.
• Cache and continue. Clients should keep serving with last known good configuration when discovery is unavailable.
• Keep an independent fallback for discovery on critical paths, and rehearse switching to it.
• Monitor discovery directly, as Spotify's follow-ups did, so a discovery problem is identified as one quickly rather than inferred from widespread symptoms.