Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Post Mortem of Google Outage on 14 December 2020

status.cloud.google.com

Post reply on HN
Print view View original

71–80 of 209 posts

Re: Post Mortem of Google Outage on 14 December 2020

#71

Post by azornathogron »

evil-olive wrote: post #57
physicsgraph wrote: post #11

My favorite quote: "prevent fast implementation of global changes." This is how large organizations become slower compared to small "nimble" companies. Everyone fails, but the noticeability of failure incentivizes large organizations to be more careful.

Slow rollouts can be a double-edged sword, too: > a change was made in October to register the User ID Service with the new quota system, but parts of the previous quota system were left in place which incorrectly reported the usage for the User ID Service as 0. An existing grace period on enforcing quota restrictions delayed the impact, which eventually expired, triggering automated quota systems to decrease the quo…

"Slow" isn't really a precise enough descriptor.

You need gradual rollouts. In particular, you need rollouts where the behavior of your system changes gradually as you apply your change to more of your instances/zones/whatever-rollout-unit. And the right speed is whatever speed gives you enough time to detect a problem and stop the rollout while the damage is still small enough to be "acceptable". With "acceptable" determined by the needs of your service (but if you say "no damage is ever acceptable" then I have some bad news for you).

Grace periods don't give you gradual rollouts like this; that's not their purpose. And I agree, grace periods can be a double edged sword for the reason you mention.

Re: Post Mortem of Google Outage on 14 December 2020

#72

Post by mehrdadn »

b112 wrote: post #70
mehrdadn wrote: post #9

I'm skimming this but so far I still don't understand how any kind of authentication failure can or should lead to an SMTP server returning "this address doesn't exist". Does anyone see an explanation? Edit 1: Oh wow I didn't even realize there were multiple incidents. Thanks! Edit 2 (after reading the Gmail post-mortem): AH! It was a messed up domain name! When this event happened, I said senders need to avoid takin…

This is how mail works. Follow the RFC or you're wrong. https://tools.ietf.org/html/rfc5321 I haven't read the article, but SMTP response codes are very specific. If there were SMTP response codes in the 5xx range, that's a perm failure, end of story. Temp fail messages, 4xx, can be used for 'try later'. That's how SMTP works. What you are suggesting would break email.

> What you are suggesting would break email.

No it would not. Firstly, because you would just rejected twice; the sky wouldn't fall down. Secondly, because the RFC in fact permits this too; it's within spec even if you want to follow it blindly to the letter with zero consideration for the context. People already litigated this in the earlier discussion and there's no point rehashing it so I'll just leave it at this.

Re: Post Mortem of Google Outage on 14 December 2020

#73

Post by nullifidian »

"I hated that app (vscode) on my last laptop, it was very slow and bloated, and I've been considering switching to something else. Now I don't care enough to see whether it's actually optimized or not, it's faster than my brain and that's quite enough."

That's the sad part. That's how software becomes slower and slower with every year.

Re: Post Mortem of Google Outage on 14 December 2020

#74

Post by ahupp »

jiggawatts wrote: post #63

I've been bitten by quota grace periods before, they're the "buffer bloat" of cloud platform management systems. You build something, it seems fine, and then a month later it keels over. My tiny disaster was caused by Amazon EFS. They provide a performance quota grace period of a month, during which all tests passed with flying colours. Turns out that EFS rations out IOPS per GB of stored data, and this particular ap…

I ran into this same issue with GCP while using the boot disk for some caches. In this case the grace period is a few minutes before they throttle. It was quite a pain to track down.

Re: Post Mortem of Google Outage on 14 December 2020

#75

Post by asdfasgasdgasdg »

x87678r wrote: post #49

I'm curious about big outages like this in big internet corps. Does anyone know if SREs in Europe fixed the problem or it relied on people in Mountain View? When an outage this big hits do devs get involved? I've worked as an SRE and it sucks to be fixing developer's bugs in the middle of the night.

I assume this is in the SRE book, but a tier one product like the identity service will have global SRE coverage (i.e. at least three SRE teams so that there is always an SRE group for whom it is daytime holding the pager). Devs are often involved in diagnosis, but are less often required for mitigation, as the mitigation is almost always to revert whatever change caused the problem. This is a simplification of course, but it gives an idea of the general pattern.

Re: Post Mortem of Google Outage on 14 December 2020

#76

Post by mulcahey »

physicsgraph wrote: post #11

My favorite quote: "prevent fast implementation of global changes." This is how large organizations become slower compared to small "nimble" companies. Everyone fails, but the noticeability of failure incentivizes large organizations to be more careful.

"Move fast and break things!"

...

"Move fast! ...with stable infrastructure!" [1]

[1] https://www.cnet.com/news/zuckerberg-move-fast-and-break-thi...

Re: Post Mortem of Google Outage on 14 December 2020

#77

Post by vlovich123 »

crmd wrote: post #69
jiggawatts wrote: post #63

I've been bitten by quota grace periods before, they're the "buffer bloat" of cloud platform management systems. You build something, it seems fine, and then a month later it keels over. My tiny disaster was caused by Amazon EFS. They provide a performance quota grace period of a month, during which all tests passed with flying colours. Turns out that EFS rations out IOPS per GB of stored data, and this particular ap…

The linear IOP density model seems clever and logical but is a huge source of headaches because it includes a patently false assumption that IOPs scale in proportion to growth in object size. Performance quotas should be assigned at the object level (block device, file system, bucket) regardless of size.

It’s not because the underlying storage is 1 TiB disk, so you’re getting a fixed allocation that’s shared among many other 50mb/s clients (ie assuming flash and overall transfer rates of ~2gigabytes/, that lets them coex roughly 40 customers on 1 machine without over subscribing)?

Isn’t that kind of pricing mode about the only one that would be feasible to implement to make this cost effective? How are you thinking it should work? Fixed cost per block transferred?

Re: Post Mortem of Google Outage on 14 December 2020

#78

Post by b112 »

mehrdadn wrote: post #72
b112 wrote: post #70

Earlier quoted context omitted.

This is how mail works. Follow the RFC or you're wrong. https://tools.ietf.org/html/rfc5321 I haven't read the article, but SMTP response codes are very specific. If there were SMTP response codes in the 5xx range, that's a perm failure, end of story. Temp fail messages, 4xx, can be used for 'try later'. That's how SMTP works. What you are suggesting would break email.

> What you are suggesting would break email. No it would not. Firstly, because you would just rejected twice; the sky wouldn't fall down. Secondly, because the RFC in fact permits this too; it's within spec even if you want to follow it blindly to the letter with zero consideration for the context. People already litigated this in the earlier discussion and there's no point rehashing it so I'll just leave it at this.

Rejected twice? When do you retry? 5 seconds later? 5 minutes? An hour? When is the message requeued to send? When do you stop trying, when the remote side has already said "Hey, this user has no account here!".

Meanwhile, the end user sent a message to bod@ by accident, instead of bob@, and your mail server keeps retrying to send mail, even when the remote mail server said "Hey! That account doesn't exist!".

It's a typo, but you've decided to 'make things better', so now, until that bounce happens, the end user won't find out they made that mistake.

Things work as they do for a reason, and the people saying 5xx means bounce! are right.

Re: Post Mortem of Google Outage on 14 December 2020

#79

Post by tantalor »

sleepydog wrote: post #60
blaisio wrote: post #17

Hmm one thing that jumped out at me was the organizational mistake of having a very long automated "grace period". This is actually bad system architecture. Whenever you have a timeout for something that involves a major config change like this, the timeout must be short (like less than a week). Otherwise, it is very likely people will forget about it, and it will take a while for people to recognize and fix the prob…

I agree, and even if the grace period were a good idea, enforcement should have slowly ratcheted up over the grace period, rather than having full enforcement immediately after it expired.

This is also called a "time bomb". It's a bad thing.

Re: Post Mortem of Google Outage on 14 December 2020

#80

Post by inopinatus »

simon1573 wrote: post #68
inopinatus wrote: post #44

Earlier quoted context omitted.

It doesn’t have to be this way, but that’s partly a matter of culture. By aspiring to present/think/act as a monoplatform, Google risks substantially increasing the blast radius of individual component failure. A global quota system mediating every other service sounds both totally on brand, and also the antithesis of everything I learned about public cloud scaling at AWS. There we made jokes, that weren’t jokes, abo…

Do you blog? I really enjoyed reading that.

Thanks. I'm more a forum-dweller when it comes to self-expression. There's an obvious .org but you'll be sorely disappointed, unless you're looking for arcane and infrequent Ruby/Rails tips.
Post reply on HN