What caused the Google Cloud Storage outage on March 12, 2019?

According to Google's incident report, a configuration change made by Google's SREs had a side effect of overloading a key part of the internal blob storage system used to look up the location of blob data. The overload triggered a cascading failure that raised error rates across Cloud Storage and other services.

Which services were affected by the March 2019 Google storage incident?

Google's report lists Google Cloud Storage, Stackdriver Monitoring, and App Engine's Blobstore API and deployments, among others. User-facing services such as Gmail, Google Drive, and Google Photos also saw elevated error rates, with impact reduced by caching and redundancy in those services.

What is a cascading failure?

It is a failure that feeds itself. An overloaded component fails requests, clients retry, and the extra load keeps it overloaded or spreads overload to other components. Removing the original trigger is often not enough to recover; load must be reduced until the system stabilises, then restored gradually.

How do you protect a metadata service from overload?

Give it generous capacity headroom, reject excess requests early and cheaply with load shedding, limit client retries with budgets and backoff, isolate it by region where possible, stage configuration changes with canaries and automatic rollback, and rehearse recovery, including how to reduce incoming traffic deliberately and restore it gradually.

Google Cloud Storage, March 2019: A Configuration Change Overloads Metadata

Per Google's incident report, on March 12, 2019, a configuration change by Google's SREs overloaded the part of its internal blob storage system that looks up where data is stored. The cascading failure raised error rates across Cloud Storage and other services for over four hours. Recovery required stopping the change and manually reducing traffic.

What Google reported

Google's incident report for Cloud Storage incident 19002 states that the disruption began on March 12, 2019 at 18:40 and ended at 22:50 US/Pacific, a duration of four hours and ten minutes. Google's SREs had made a configuration change which had a side effect of overloading a key part of the system for looking up the location of blob data, and the overload triggered a cascading failure in the internal blob storage service.

The report gives error rates: the internal blob service averaged 20 percent errors with a peak of 31 percent, and Google Cloud Storage saw an average error rate of 4.8 percent along with elevated long-tail latency. App Engine's Blobstore API and deployments, and Stackdriver Monitoring, were also affected. User-facing services including Gmail, Google Drive, and Google Photos saw elevated errors, with impact reduced by their caching and redundancy.

SREs were alerted at 18:56. They stopped the job making configuration changes and manually reduced traffic levels to let the system recover. Google committed to improving regional isolation, the speed of provisioning resources, validation of configuration changes, and graceful load shedding in the metadata system.

Why metadata layers are the weak point

Large storage systems separate data from metadata: the index of what exists and where it lives. Data is spread across many machines; the metadata lookup path is comparatively small, centralised, and consulted on almost every operation.

That shape makes it a natural bottleneck. If the lookup layer slows or fails, reads and writes fail even though the underlying data is intact and the storage nodes are healthy. And because many services build on the same storage, a problem there appears simultaneously across products that seem unrelated.

Why reverting was not enough

The report describes a cascading failure, and the recovery steps show why that matters: stopping the configuration job was followed by manually reducing traffic. In a cascading failure, an overloaded component fails requests, clients retry, and the added load keeps the component overloaded even after the original trigger is removed. The system is stable in its failed state.

Breaking out of that state requires shedding load until the component can recover, then restoring traffic gradually. Systems that shed load gracefully on their own, rejecting excess work cheaply and early, recover without manual intervention. That is why graceful load shedding appears among Google's remediations. Restoring traffic too quickly after recovery can push the system straight back into overload, so the ramp back up needs the same care as the ramp down.

Configuration changes as the trigger

The trigger was a configuration change, not a code deployment. Configuration changes often move faster and receive less scrutiny than code, yet they can alter load, routing, and capacity just as dramatically. Google's stated follow-up, software measures to prevent configuration changes that overload key parts of the system, reflects a lesson that recurs across many public postmortems: configuration deserves the same staged rollout, validation, and automatic halting as code.

Lessons for your own systems

Identify your metadata tiers. Indexes, catalogues, routing tables, and lock services that every request touches deserve extra capacity headroom and protection.
Stage configuration changes with the same canarying and automatic rollback as code, and watch load on critical tiers during rollout.
Build graceful load shedding into critical components so overload produces fast rejections instead of collapse.
Budget retries so clients cannot sustain a cascading failure.
Isolate by region where possible, so a failure in one place does not become global.
Rehearse recovery from overload, including how to reduce traffic deliberately and restore it in steps.

None of these would make such an incident impossible. Together they shrink its size, shorten its duration, and make recovery a practised procedure rather than an improvisation.

Frequently asked questions

What caused the Google Cloud Storage outage on March 12, 2019?: According to Google's incident report, a configuration change made by Google's SREs had a side effect of overloading a key part of the internal blob storage system used to look up the location of blob data. The overload triggered a cascading failure that raised error rates across Cloud Storage and other services.
Which services were affected by the March 2019 Google storage incident?: Google's report lists Google Cloud Storage, Stackdriver Monitoring, and App Engine's Blobstore API and deployments, among others. User-facing services such as Gmail, Google Drive, and Google Photos also saw elevated error rates, with impact reduced by caching and redundancy in those services.
What is a cascading failure?: It is a failure that feeds itself. An overloaded component fails requests, clients retry, and the extra load keeps it overloaded or spreads overload to other components. Removing the original trigger is often not enough to recover; load must be reduced until the system stabilises, then restored gradually.
How do you protect a metadata service from overload?: Give it generous capacity headroom, reject excess requests early and cheaply with load shedding, limit client retries with budgets and backoff, isolate it by region where possible, stage configuration changes with canaries and automatic rollback, and rehearse recovery, including how to reduce incoming traffic deliberately and restore it gradually.

Key takeaways

Per Google, a configuration change overloaded the metadata lookup layer of its blob storage service.
The internal blob service averaged 20 percent errors, peaking at 31 percent, per the report.
Metadata tiers are small, central, and on every request's path, so overload spreads widely.
Recovery required reducing load, not just reverting the change, which is typical of cascading failure.
Google's remediations focused on isolation, change validation, provisioning speed, and load shedding.