Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Post Mortem of Google Outage on 14 December 2020

status.cloud.google.com

Post reply on HN
Print view View original

61–70 of 209 posts

Re: Post Mortem of Google Outage on 14 December 2020

#61

Post by NoodleIncident »

maxander wrote: post #53
cpncrunch wrote: post #39

Earlier quoted context omitted.

Indeed, it seems to be a completely unrelated issue. Two major Google outages on the same day for different reasons.

The odds that they would happen simultaneously if they’re completely unrelated seem astronomically small, certainly? Both are noted as being related to “ongoing migrations,” though AFAICT not related ones. I would bet there’s a human factor connection- e.g., the day before there was a big meeting where a higher-level management gave multiple ops teams go-ahead on their respective plans, resulting in a multiple potent…

I think it's as simple as a case of the Mondays; you wouldn't roll out a migration like that on a Friday or the weekend, and rolling it out on a Monday gives you the least chances of problem occurring on those dates.

Re: Post Mortem of Google Outage on 14 December 2020

#62

Post by NoodleIncident »

pdkl95 wrote: post #41
azornathogron wrote: post #5

Earlier quoted context omitted.

Here is the incident report for the Gmail problem: https://static.googleusercontent.com/media/www.google.com/en... It is linked from the Google Workspace status page here: https://www.google.com/appsstatus#hl=en-GB&v=issue&sid=1&iid...

> A configuration change during this migration shifted the formatting behavior of a service option so that it incorrectly provided an invalid domain name, instead of the intended "gmail.com" domain name, to the Google SMTP inbound service. Wow... how was this even possible? Did they do any testing whatsoever before migrating the live production system? They misformatting the domain name should have broken even basic…

The nature of configuration is that it's different for prod and your testing environments. It doesn't make it impossible to test your prod config changes, but it's not that simple either.

Re: Post Mortem of Google Outage on 14 December 2020

#63

Post by jiggawatts »

I've been bitten by quota grace periods before, they're the "buffer bloat" of cloud platform management systems. You build something, it seems fine, and then a month later it keels over.

My tiny disaster was caused by Amazon EFS. They provide a performance quota grace period of a month, during which all tests passed with flying colours. Turns out that EFS rations out IOPS per GB of stored data, and this particular application stored only a few MB at the time, because it was brand new and hadn't accumulated anything yet. I had a very angry customer calling up asking me to please explain why they were seeing an average of 0.1 IOPS...

From: https://docs.aws.amazon.com/efs/latest/ug/performance.html

"The baseline rate is 50 MiB/s per TiB of storage (equivalently, 50 KiB/s per GiB of storage).

AWS now provides a performance floor of 1 MiB/s, but at the time there was no floor. If I remember correctly, this application had something like 2 MIB of data, which was constantly being updated by various processes, so there was no quota being accumulated. The system performance went from something like 1 Gbps to 100 bytes per second instantly. It took 10 seconds for a 1 KiB I/O to complete. Fun times, fun times...

Re: Post Mortem of Google Outage on 14 December 2020

#64

Post by NoodleIncident »

blaisio wrote: post #17

Hmm one thing that jumped out at me was the organizational mistake of having a very long automated "grace period". This is actually bad system architecture. Whenever you have a timeout for something that involves a major config change like this, the timeout must be short (like less than a week). Otherwise, it is very likely people will forget about it, and it will take a while for people to recognize and fix the prob…

What's insane to me is that a grace period is built in, but that the mere fact that this grace protection is active isn't a giant neon sign on their dashboards and alerts. I do see how it could slip through the cracks, since it was the reported usage that was wrong, not the quota itself.

Re: Post Mortem of Google Outage on 14 December 2020

#66

Post by ffggvv »

physicsgraph wrote: post #11

My favorite quote: "prevent fast implementation of global changes." This is how large organizations become slower compared to small "nimble" companies. Everyone fails, but the noticeability of failure incentivizes large organizations to be more careful.

its a paradox because they should be slower when they have so many things depending on them and they can't afford to fail. so being quicker would actually be dumb of them.

Re: Post Mortem of Google Outage on 14 December 2020

#67

Post by sdwvit »

jrockway wrote: post #18
mehrdadn wrote: post #9

I'm skimming this but so far I still don't understand how any kind of authentication failure can or should lead to an SMTP server returning "this address doesn't exist". Does anyone see an explanation? Edit 1: Oh wow I didn't even realize there were multiple incidents. Thanks! Edit 2 (after reading the Gmail post-mortem): AH! It was a messed up domain name! When this event happened, I said senders need to avoid takin…

Different outage. The Gmail postmortem is linked in another thread, but the gist was that "gmail.com" is a configuration value that can be changed at runtime, and someone changed the configuration. Thus, *@gmail.com stopped being a valid address, and they returned "that mailbox is unavailable". If you don't want to scroll to the other thread, here's the postmortem: https://static.googleusercontent.com/media/www.googl…

What strikes me is “ On 2020-12-14, Google Engineering began receiving internal user reports of delivery errors. A code change which updated the configuration was identified as having a compatibility issue. Believing that this was the root cause, Google Engineering reverted the code change, which resolved the immediate issue” So they didn’t investigate, just believed that this is an issue. That’s crazy. It tells me about how slow processes can be at google

Re: Post Mortem of Google Outage on 14 December 2020

#68

Post by simon1573 »

inopinatus wrote: post #44
physicsgraph wrote: post #11

My favorite quote: "prevent fast implementation of global changes." This is how large organizations become slower compared to small "nimble" companies. Everyone fails, but the noticeability of failure incentivizes large organizations to be more careful.

It doesn’t have to be this way, but that’s partly a matter of culture. By aspiring to present/think/act as a monoplatform, Google risks substantially increasing the blast radius of individual component failure. A global quota system mediating every other service sounds both totally on brand, and also the antithesis of everything I learned about public cloud scaling at AWS. There we made jokes, that weren’t jokes, abo…

Do you blog? I really enjoyed reading that.

Re: Post Mortem of Google Outage on 14 December 2020

#69

Post by crmd »

jiggawatts wrote: post #63

I've been bitten by quota grace periods before, they're the "buffer bloat" of cloud platform management systems. You build something, it seems fine, and then a month later it keels over. My tiny disaster was caused by Amazon EFS. They provide a performance quota grace period of a month, during which all tests passed with flying colours. Turns out that EFS rations out IOPS per GB of stored data, and this particular ap…

The linear IOP density model seems clever and logical but is a huge source of headaches because it includes a patently false assumption that IOPs scale in proportion to growth in object size. Performance quotas should be assigned at the object level (block device, file system, bucket) regardless of size.

Re: Post Mortem of Google Outage on 14 December 2020

#70

Post by b112 »

mehrdadn wrote: post #9

I'm skimming this but so far I still don't understand how any kind of authentication failure can or should lead to an SMTP server returning "this address doesn't exist". Does anyone see an explanation? Edit 1: Oh wow I didn't even realize there were multiple incidents. Thanks! Edit 2 (after reading the Gmail post-mortem): AH! It was a messed up domain name! When this event happened, I said senders need to avoid takin…

This is how mail works. Follow the RFC or you're wrong.

https://tools.ietf.org/html/rfc5321

I haven't read the article, but SMTP response codes are very specific. If there were SMTP response codes in the 5xx range, that's a perm failure, end of story. Temp fail messages, 4xx, can be used for 'try later'.

That's how SMTP works. What you are suggesting would break email.

Post reply on HN