Post Mortem of Google Outage on 14 December 2020

Post by pram » Sat, Dec 19, 2020, 12:42 AM UTC

vlovich123 wrote: post #77

crmd wrote: post #69

Earlier quoted context omitted.

The linear IOP density model seems clever and logical but is a huge source of headaches because it includes a patently false assumption that IOPs scale in proportion to growth in object size. Performance quotas should be assigned at the object level (block device, file system, bucket) regardless of size.

It’s not because the underlying storage is 1 TiB disk, so you’re getting a fixed allocation that’s shared among many other 50mb/s clients (ie assuming flash and overall transfer rates of ~2gigabytes/, that lets them coex roughly 40 customers on 1 machine without over subscribing)? Isn’t that kind of pricing mode about the only one that would be feasible to implement to make this cost effective? How are you thinking i…

Post by rachelbythebay » Sat, Dec 19, 2020, 12:48 AM UTC

jeffbee wrote: post #22

azornathogron wrote: post #5

Earlier quoted context omitted.

Here is the incident report for the Gmail problem: https://static.googleusercontent.com/media/www.google.com/en... It is linked from the Google Workspace status page here: https://www.google.com/appsstatus#hl=en-GB&v=issue&sid=1&iid...

Damn, that's awful. Should lead to a deep questioning of people who claim to be migrating your junk to "best practices" if the old config system worked fine for 15 years and the new one caused a massive dataloss outage on Day 1.

Post by herf » Sat, Dec 19, 2020, 12:54 AM UTC

Post by ridaj » Sat, Dec 19, 2020, 1:06 AM UTC

inopinatus wrote: post #88

ridaj wrote: post #87

Earlier quoted context omitted.

Well, GCP followed that principle as their service account auth mechanisms did not fall over. So if you were to compare with AWS, it looks like something similar was happening. The infrastructure running the auth service, like any service, is going to have a quota system, whether it's global or not. The lesson learned might be different if it wasn't global ("prevent fast changes to the quota system for the auth servi…

> The infrastructure running the auth service, like any service, is going to have a quota system, whether it's global or not. Don’t assume that is the case. That’s exactly the kind of cultural assumption I’m speaking of. Case in point, I routinely run services without quotas or caps and what have you, scale out for load, and alarm on runaway usage, not service unavailable or quota exceeded. I’d rather take the hit th…

Post by freeone3000 » Sat, Dec 19, 2020, 1:07 AM UTC

vlovich123 wrote: post #92

freeone3000 wrote: post #83

Earlier quoted context omitted.

Fixed cost per IOPS allocated. Essentially, the same thing as before, but without the necessity of you storing large blank objects.

How would scheduling of concurrent I/O workloads work? If I’m paying some price per 1k IOPs and my service gets a spike, won’t I greedily take out the other services running on the same machine rather than getting throttled? Doesn’t this also penalize workloads that do lots of small I/Os rather than a few big ones even if the amount transferred is the same?