Can you imagine the stress of being on-call for this? Shudder!
Post Mortem of Google Outage on 14 December 2020
121–130 of 209 posts
Re: Post Mortem of Google Outage on 14 December 2020
#122I've been bitten by quota grace periods before, they're the "buffer bloat" of cloud platform management systems. You build something, it seems fine, and then a month later it keels over. My tiny disaster was caused by Amazon EFS. They provide a performance quota grace period of a month, during which all tests passed with flying colours. Turns out that EFS rations out IOPS per GB of stored data, and this particular ap…
Even worse are undocumented quotas. Probably not quite the same thing, but I've had such an issue with a SaaS API for extracting data for a cloud system. It has an unpublished limit on queries. Writing a library of code to use it, I was fine. All tests passed. Running a full day's data pull & integration had sporadic failure: It would run w/o issue for half the job, then have seeming random errors on some requests. I…
I had a similar experience with a SaaS. They let us run fine for months, then started degrading our service. When we contacted them, they tried to convince us we had a bug in our implementation. No, we confirmed we were using their service exactly as we intended to. They seemed to have designed their service with a particular use case in mind, and ours wasn't it. We tried multiple times to explain our use case and why their service was useful for us, because we didn't want them to think we were using the service in bad faith, but if they understood, they never let on. After much grumbling, they would lift the restrictions on us, but apparently not far, because we'd run into them again weeks or months later as our traffic increased. They were obviously unhappy with the way we were using their service and determined to cause issues for us, so we gave up and stopped using them.
Re: Post Mortem of Google Outage on 14 December 2020
#123Earlier quoted context omitted.
It doesn’t have to be this way, but that’s partly a matter of culture. By aspiring to present/think/act as a monoplatform, Google risks substantially increasing the blast radius of individual component failure. A global quota system mediating every other service sounds both totally on brand, and also the antithesis of everything I learned about public cloud scaling at AWS. There we made jokes, that weren’t jokes, abo…
I often hear 'aim for elimination of global dependencies', but the reality is that there is no way around global dependencies. AWS STS or IAM is just as global as google's. The difference is that google more often builds with some form of guaranteed read-after-write consistency, while AWS is more often 'fail open'. For example, if you remove a permission from a user in GCP, you are guaranteed consistency within 7 min…
Side note: AWS STS has had regional endpoints for years. The global endpoint is vestigial at this point. I didn't glean anything special about Google's endpoint that requires it to be globalized like this, but I can't really criticize it without knowing the details.
Re: Post Mortem of Google Outage on 14 December 2020
#124Well I guess the "Code Purple" got its own "Code Red"[1]
The take away for me here is that maximizing resource utilization continues to be a hard problem and as you get better at it, the margin for errors is smaller and smaller.
[1] Sorry its an inside Google joke.
Re: Post Mortem of Google Outage on 14 December 2020
#125I've been bitten by quota grace periods before, they're the "buffer bloat" of cloud platform management systems. You build something, it seems fine, and then a month later it keels over. My tiny disaster was caused by Amazon EFS. They provide a performance quota grace period of a month, during which all tests passed with flying colours. Turns out that EFS rations out IOPS per GB of stored data, and this particular ap…
Even worse are undocumented quotas. Probably not quite the same thing, but I've had such an issue with a SaaS API for extracting data for a cloud system. It has an unpublished limit on queries. Writing a library of code to use it, I was fine. All tests passed. Running a full day's data pull & integration had sporadic failure: It would run w/o issue for half the job, then have seeming random errors on some requests. I…
There are some "fun" horror stories of people hitting the Azure Resource Manager API quota limits. At least one of them was linked to some sort of shared service principal for the client org, so if anyone hit it, the whole organisation could get locked out.
It would first start to throw HTTP 429 codes occasionally, then you would get locked out for exponentially increasing times, up to two weeks or something absurd like that.
Even if you called support, there was not a lot they could do, because the rate limiter is a low-level thing built into an internal load balancer somewhere, and it is difficult to override it for a single account.
The worst part was that you could hit the limit while not actually taking any action! If you just had certain Azure Portal screens open, the JavaScript would refresh things in the background, constantly consuming the API rate limit quota. Some screens do so many calls that you'll hit the rate limit in a matter of minutes if you leave your browser on those. If you fail to close all browser sessions of all admins right away, you can lock yourself out for days.
Re: Post Mortem of Google Outage on 14 December 2020
#126I've been bitten by quota grace periods before, they're the "buffer bloat" of cloud platform management systems. You build something, it seems fine, and then a month later it keels over. My tiny disaster was caused by Amazon EFS. They provide a performance quota grace period of a month, during which all tests passed with flying colours. Turns out that EFS rations out IOPS per GB of stored data, and this particular ap…
Even worse are undocumented quotas. Probably not quite the same thing, but I've had such an issue with a SaaS API for extracting data for a cloud system. It has an unpublished limit on queries. Writing a library of code to use it, I was fine. All tests passed. Running a full day's data pull & integration had sporadic failure: It would run w/o issue for half the job, then have seeming random errors on some requests. I…
Ooh boy. Undocumented limitations caused major problems with us with AWS SQS. It is now in the documentation (maybe because I complained to our AWS rep), but SQS has pool of 20k messages that it will pull from when serving requests. If you are using a FIFO queue and you have 20k messages with the same message group ID in that buffer, then you are unable to process any other message regardless of how large the queue is. It caused multiple severe degradations until we rearchitected how messages are grouped. If only it had been in the documentation from day 1.
Gotta love it!
Re: Post Mortem of Google Outage on 14 December 2020
#127Earlier quoted context omitted.
The "pain to track down" is important because nothing failed . I didn't get an alert. There was nothing in the logs. There wasn't anything in the portal to indicate that something had changed. Everything was up and responding , just really, really slowly. The application wasn't even timing out, because the EFS share itself was responding to TCP ACKs instantly, and even the timeouts at the NFS protocol layer weren't b…
To be fair, there is a cloud watch metric you could have set an alert on: "PercentIOLimit"
Ahhh... the joy of enterprise monitoring systems that do exactly nothing by default, and are very helpful in avoiding any further recurrences of one-time issues. At a nominal fee, of course.
The golden rule of both backups and monitoring is: There are no time machines.
It's not helpful to find out after the fact that a default-off alert or metric threshold alarm could have avoided the issue. It's not helpful to blame the user for not knowing every one of thousands of metrics they "should" be monitoring. How would they know until they get burnt at least once?
Even if they do get burnt, how would they know which of the metrics they weren't capturing could have been useful it was was captured?
That's not a rhetorical question!
Fundamentally the issue is this: Practically no enterprise monitoring system stores data efficiently enough to capture all metrics, so instead they simply... don't.
Instead, these "solutions" trade a moderately difficult storage compression problem at the service provider end for a physically impossible time travel problem on the consumer end.
Just blame the user for not knowing ahead what disasters they will face! Job done! No need to figure out columnar compression, that would take actual engineering work for a couple of guys. But why bother when it's soooo much easier to just dump some JSON into a storage account or S3 bucket and bill the customer for every metric. Mmm... dollars per metric per month. That's the ticket to a nice robust revenue stream!
Apologies if I sound salty, but I've traced the root cause of outage after outage back to lazy vendors writing MVP monitoring systems that do literally nothing useful out of the box. These vendors simply refuse to store data efficiently enough to capture all relevant metrics to that I can check what happened without needing a TARDIS. Why would they when monitoring is a revenue stream that they measure in gigabytes?
PS: An ordinary Windows desktop has on the order of 10,000 to 50,000 performance counter metrics that it tracks. However, with even light compression, that's barely a few gigabytes for a year of logging every metric every second. I've written code to do this personally. Name me a cloud vendor that can approach this within an order of magnitude without an eye-watering bill every month.
Re: Post Mortem of Google Outage on 14 December 2020
#128Earlier quoted context omitted.
It doesn’t have to be this way, but that’s partly a matter of culture. By aspiring to present/think/act as a monoplatform, Google risks substantially increasing the blast radius of individual component failure. A global quota system mediating every other service sounds both totally on brand, and also the antithesis of everything I learned about public cloud scaling at AWS. There we made jokes, that weren’t jokes, abo…
I often hear 'aim for elimination of global dependencies', but the reality is that there is no way around global dependencies. AWS STS or IAM is just as global as google's. The difference is that google more often builds with some form of guaranteed read-after-write consistency, while AWS is more often 'fail open'. For example, if you remove a permission from a user in GCP, you are guaranteed consistency within 7 min…
> In general, AWS more often shifts the harder parts of global distributed systems onto their customers, rather than solving them for their customers, like GCP does.
Choice of language in representing this is rather telling, because AWS can (and does) pitch this as a strength, viz. that regionalisation helps customers (especially, significantly, bigco enterprise customers) reason about the possible failure modes, and thereby contain the blast radius of component failure.
They'd never comment on competitors in public, but the clear implication is that apparently global services merely gloss over the risks, they don't resolve them, and eventually it'll blow up in your face, or someone's face at least.
> there is no way around global dependencies
This sounds more like a challenge than an assertion. In my very long experience of tech, anyone who ever said, "you can't do that", eventually ate their hat.
Re: Post Mortem of Google Outage on 14 December 2020
#129I've been bitten by quota grace periods before, they're the "buffer bloat" of cloud platform management systems. You build something, it seems fine, and then a month later it keels over. My tiny disaster was caused by Amazon EFS. They provide a performance quota grace period of a month, during which all tests passed with flying colours. Turns out that EFS rations out IOPS per GB of stored data, and this particular ap…
My unexpected AWS quota story: AWS Batch. Using managed instances, just asking AWS to provision “suitable and available”. Daily jobs (docker containers) ran fine for the first 6 months. Eventually started stalling and failing with DockerTimeout. They would fail in batches of 100s and then at some point start working again.
It took several hours and some back and forth with support to realize that the burst IOPS quota of the provisioned underlying EBS disks on the EC2 instances forming the ECS cluster had been depleted so disk performance completely tanked to the point the docker agent couldn’t be reached for 4 minutes.
And here I thought the whole point was that these things would be abstracted away...
Re: Post Mortem of Google Outage on 14 December 2020
#130Earlier quoted context omitted.
I think it's as simple as a case of the Mondays; you wouldn't roll out a migration like that on a Friday or the weekend, and rolling it out on a Monday gives you the least chances of problem occurring on those dates.
A Monday rollout sounds horrible. Probably the worst day for a rollout.
Why is that?