Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Post Mortem of Google Outage on 14 December 2020

status.cloud.google.com

Post reply on HN
Print view View original

81–90 of 209 posts

Re: Post Mortem of Google Outage on 14 December 2020

#81

Post by notacoward »

mulcahey wrote: post #76
physicsgraph wrote: post #11

My favorite quote: "prevent fast implementation of global changes." This is how large organizations become slower compared to small "nimble" companies. Everyone fails, but the noticeability of failure incentivizes large organizations to be more careful.

"Move fast and break things!" ... "Move fast! ...with stable infrastructure!" [1] [1] https://www.cnet.com/news/zuckerberg-move-fast-and-break-thi...

Had the same thought. Saw this same scenario play out many times between services at FB, and I'm still really not sure there's a good "one size fits all" answer either there or at peer companies like Google. For every "just do X" I've seen here I could probably identify the incident where that fix led to or exacerbated a different outage. Sometimes teams don't collaborate well, and that requires a specific fix beyond outsiders' view instead of more platitudes.

Re: Post Mortem of Google Outage on 14 December 2020

#82

Post by tpmx »

Privacy846 wrote: post #54
tpmx wrote: post #3

Thought: There should be an official Google status dashboard for "free" (paid for via personal ad targeting metadata) services like Search, Gmail, Drive, etc. With postmortems, too. Probably won't happen unless it's mandated by law, though.

Keep your scare quotes.

I kinda thought the sentence within the parentheses explained it all.

Re: Post Mortem of Google Outage on 14 December 2020

#83

Post by freeone3000 »

vlovich123 wrote: post #77
crmd wrote: post #69

Earlier quoted context omitted.

The linear IOP density model seems clever and logical but is a huge source of headaches because it includes a patently false assumption that IOPs scale in proportion to growth in object size. Performance quotas should be assigned at the object level (block device, file system, bucket) regardless of size.

It’s not because the underlying storage is 1 TiB disk, so you’re getting a fixed allocation that’s shared among many other 50mb/s clients (ie assuming flash and overall transfer rates of ~2gigabytes/, that lets them coex roughly 40 customers on 1 machine without over subscribing)? Isn’t that kind of pricing mode about the only one that would be feasible to implement to make this cost effective? How are you thinking i…

Fixed cost per IOPS allocated. Essentially, the same thing as before, but without the necessity of you storing large blank objects.

Re: Post Mortem of Google Outage on 14 December 2020

#84

Post by paxys »

blaisio wrote: post #17

Hmm one thing that jumped out at me was the organizational mistake of having a very long automated "grace period". This is actually bad system architecture. Whenever you have a timeout for something that involves a major config change like this, the timeout must be short (like less than a week). Otherwise, it is very likely people will forget about it, and it will take a while for people to recognize and fix the prob…

We once found a very annoying bug which was caused because someone set a feature flag to a tiny rollout % and then left the company without updating it. It sat that way for 2 years before someone finally noticed.

Re: Post Mortem of Google Outage on 14 December 2020

#85

Post by paxys »

tpmx wrote: post #3

Thought: There should be an official Google status dashboard for "free" (paid for via personal ad targeting metadata) services like Search, Gmail, Drive, etc. With postmortems, too. Probably won't happen unless it's mandated by law, though.

Free and paid versions of Google services share pretty much the entire stack, so the Google Workspace (previously G Suite) status page works just fine – https://www.google.com/appsstatus#hl=en&v=status.

Re: Post Mortem of Google Outage on 14 December 2020

#86

Post by joshuamorton »

tpolm wrote: post #56
cpncrunch wrote: post #45

Earlier quoted context omitted.

>both of those migrations failed with symptoms suggesting that whoever was performing them did not have deep understanding of systems architecture or safety practices and there was no one to stop them from failing. Can any single person at Google have a full understanding of all the dependencies for even a single system? I have no idea, as I've never worked there, but I would imagine that there is a lot of complexity…

somehow they managed to build complex systems like gmail, continuously develop new features there and not have massive outages due to "migrations" - suggests that something that they were doing right, they are no longer able to do

A single event is not data.

Re: Post Mortem of Google Outage on 14 December 2020

#87

Post by ridaj »

inopinatus wrote: post #44
physicsgraph wrote: post #11

My favorite quote: "prevent fast implementation of global changes." This is how large organizations become slower compared to small "nimble" companies. Everyone fails, but the noticeability of failure incentivizes large organizations to be more careful.

It doesn’t have to be this way, but that’s partly a matter of culture. By aspiring to present/think/act as a monoplatform, Google risks substantially increasing the blast radius of individual component failure. A global quota system mediating every other service sounds both totally on brand, and also the antithesis of everything I learned about public cloud scaling at AWS. There we made jokes, that weren’t jokes, abo…

Well, GCP followed that principle as their service account auth mechanisms did not fall over. So if you were to compare with AWS, it looks like something similar was happening.

The infrastructure running the auth service, like any service, is going to have a quota system, whether it's global or not. The lesson learned might be different if it wasn't global ("prevent fast changes to the quota system for the auth service") but the conclusion would be substantially similar - there is usually no good reason, and plenty of danger, for routine adjustments to large infrastructure to take place in a brusque manner.

That doesn't mean that non-infrastructure service needs to abide by the same rules...

Re: Post Mortem of Google Outage on 14 December 2020

#88

Post by inopinatus »

ridaj wrote: post #87
inopinatus wrote: post #44

Earlier quoted context omitted.

It doesn’t have to be this way, but that’s partly a matter of culture. By aspiring to present/think/act as a monoplatform, Google risks substantially increasing the blast radius of individual component failure. A global quota system mediating every other service sounds both totally on brand, and also the antithesis of everything I learned about public cloud scaling at AWS. There we made jokes, that weren’t jokes, abo…

Well, GCP followed that principle as their service account auth mechanisms did not fall over. So if you were to compare with AWS, it looks like something similar was happening. The infrastructure running the auth service, like any service, is going to have a quota system, whether it's global or not. The lesson learned might be different if it wasn't global ("prevent fast changes to the quota system for the auth servi…

> The infrastructure running the auth service, like any service, is going to have a quota system, whether it's global or not.

Don’t assume that is the case. That’s exactly the kind of cultural assumption I’m speaking of.

Case in point, I routinely run services without quotas or caps and what have you, scale out for load, and alarm on runaway usage, not service unavailable or quota exceeded. I’d rather take the hit than inconvenience my customers with an outage. In this frame of mind, quotas are a moral hazard, a safety barrier with a perverse disincentive.

That principle of “just run it deep” works even at global scale, right up until you run out of hardware to allocate, which is why growth logistics are a rarely-discussed but critical aspect of running a public cloud.

The core learnings become, how to factor out such situations at all. That might be through some kind of asynchronous processing (event-driven services, queues, tuplespaces), or co-operative backpressure (a la TCP/IP) and so on. Synchronous request/response state machines are absolute murder to scalability, so HTTP, especially when misappropriated as an RPC substrate, has a lot to answer for.

Re: Post Mortem of Google Outage on 14 December 2020

#89

Post by pram »

jiggawatts wrote: post #63

I've been bitten by quota grace periods before, they're the "buffer bloat" of cloud platform management systems. You build something, it seems fine, and then a month later it keels over. My tiny disaster was caused by Amazon EFS. They provide a performance quota grace period of a month, during which all tests passed with flying colours. Turns out that EFS rations out IOPS per GB of stored data, and this particular ap…

Yep this bit me in the ass too. When I first used it I was expecting something like an NFS share from a NetApp or ZFS Filer. I see they offer provisioned iops now but still, why not offer a version without all the rigmarole?

Re: Post Mortem of Google Outage on 14 December 2020

#90

Post by growse »

b112 wrote: post #78
mehrdadn wrote: post #72

Earlier quoted context omitted.

> What you are suggesting would break email. No it would not. Firstly, because you would just rejected twice; the sky wouldn't fall down. Secondly, because the RFC in fact permits this too; it's within spec even if you want to follow it blindly to the letter with zero consideration for the context. People already litigated this in the earlier discussion and there's no point rehashing it so I'll just leave it at this.

Rejected twice? When do you retry? 5 seconds later? 5 minutes? An hour? When is the message requeued to send? When do you stop trying, when the remote side has already said "Hey, this user has no account here!". Meanwhile, the end user sent a message to bod@ by accident, instead of bob@, and your mail server keeps retrying to send mail, even when the remote mail server said "Hey! That account doesn't exist!". It's a…

Generally speaking, if you're trying to write a reliable distributed system (and email is a massively distributed system), a good principle to follow is to retry on failure, no matter the failure.

Obviously there are edge cases, and obviously you don't just retry every 1ms forever, but to assume that an error that comes back from a system you've called once is both (a) the product of a 100% working non-faulty system and/or (b) not in any way a transitory issue or failure is naive. This type of thinking is how we end up with brittle systems.

A cautious implementation might see an SMTP 500 and then may just choose to try it again once or twice a little later, to see exactly how 'permanent' that failure is.

What the "but the RFC says!" people seem to forget is that the spec is only meaningful when the system is working correctly. Good engineering never assumes that.

Post reply on HN