>
[A] configuration change [...] was intended for a small number of servers in a single region. The configuration was incorrectly applied to a larger number of servers across several neighboring regions, and it caused those regions to stop using more than half of their available network capacity. The network traffic to/from those regions then tried to fit into the remaining network capacity, but it did not. The network became congested, and our networking systems correctly triaged the traffic overload and dropped larger, less latency-sensitive traffic in order to preserve smaller latency-sensitive traffic flows...> Google’s engineering teams detected the issue within seconds, but diagnosis and correction took far longer than our target of a few minutes. Once alerted, engineering teams quickly identified the cause of the network congestion, but the same network congestion which was creating service degradation also slowed the engineering teams’ ability to restore the correct configurations, prolonging the outage.
Someone forgot to classify management traffic as high-priority? Oops.
The description is vague about what devices ("servers") were misconfigured. Did someone tell all google service pods in the affected regions to restrict bandwidth by over 50%? Mentioning "server" and then talking about network congestion is confusing. How would restricted bandwidth utilization on servers cause network congestion, unless load balancers saturated the network by re-sending requests to servers because none of them were responding?