My favorite quote: "prevent fast implementation of global changes." This is how large organizations become slower compared to small "nimble" companies. Everyone fails, but the noticeability of failure incentivizes large organizations to be more careful.
Post Mortem of Google Outage on 14 December 2020
11–20 of 209 posts
Re: Post Mortem of Google Outage on 14 December 2020
#12TLDR: the team forgot to update the resource quota requirements for a critical component of Google’s authentication system while transitioning between quota systems.
That's not the TL;DR is it? It seems that the quota system detected current usage as "0", and thus adjusted the quota downwards to 0 until the Paxos leader couldn't write, which caused all of the data to become stale, which caused downstream systems to fail because they reject outdated data.
Re: Post Mortem of Google Outage on 14 December 2020
#13I'm skimming this but so far I still don't understand how any kind of authentication failure can or should lead to an SMTP server returning "this address doesn't exist". Does anyone see an explanation? Edit 1: Oh wow I didn't even realize there were multiple incidents. Thanks! Edit 2 (after reading the Gmail post-mortem): AH! It was a messed up domain name! When this event happened, I said senders need to avoid takin…
That's a different outage. SMTP 550 was the next day.
Re: Post Mortem of Google Outage on 14 December 2020
#14I'm skimming this but so far I still don't understand how any kind of authentication failure can or should lead to an SMTP server returning "this address doesn't exist". Does anyone see an explanation? Edit 1: Oh wow I didn't even realize there were multiple incidents. Thanks! Edit 2 (after reading the Gmail post-mortem): AH! It was a messed up domain name! When this event happened, I said senders need to avoid takin…
You ask the central database of accounts “does this exist?” And if it doesn’t say yes you bounce the email.
Obviously an error condition should not result in this. But complex systems. It happens.
Re: Post Mortem of Google Outage on 14 December 2020
#15I'm skimming this but so far I still don't understand how any kind of authentication failure can or should lead to an SMTP server returning "this address doesn't exist". Does anyone see an explanation? Edit 1: Oh wow I didn't even realize there were multiple incidents. Thanks! Edit 2 (after reading the Gmail post-mortem): AH! It was a messed up domain name! When this event happened, I said senders need to avoid takin…
It is not the same incident. This was for the global outage of google, not for the Gmail incident.
Re: Post Mortem of Google Outage on 14 December 2020
#16I'm skimming this but so far I still don't understand how any kind of authentication failure can or should lead to an SMTP server returning "this address doesn't exist". Does anyone see an explanation? Edit 1: Oh wow I didn't even realize there were multiple incidents. Thanks! Edit 2 (after reading the Gmail post-mortem): AH! It was a messed up domain name! When this event happened, I said senders need to avoid takin…
This is a massive distributed system.
Can't get quota? Certain pieces accidentally turn "TooManyRequests / 429s" or related semantics into 500s, 400s, 401s, etc. as you percolate upstream.
Auth is one of the most central components of any system, so there would be cascading failures everywhere.
Re: Post Mortem of Google Outage on 14 December 2020
#17Hmm one thing that jumped out at me was the organizational mistake of having a very long automated "grace period". This is actually bad system architecture. Whenever you have a timeout for something that involves a major config change like this, the timeout must be short (like less than a week). Otherwise, it is very likely people will forget about it, and it will take a while for people to recognize and fix the problem. The alternative is to just use a calendar and have someone manually flip the switch when they see the reminder pop up. Over reliance on automated timeouts like this is indicative of a badly designed software ownership structure.
Re: Post Mortem of Google Outage on 14 December 2020
#18I'm skimming this but so far I still don't understand how any kind of authentication failure can or should lead to an SMTP server returning "this address doesn't exist". Does anyone see an explanation? Edit 1: Oh wow I didn't even realize there were multiple incidents. Thanks! Edit 2 (after reading the Gmail post-mortem): AH! It was a messed up domain name! When this event happened, I said senders need to avoid takin…
Different outage. The Gmail postmortem is linked in another thread, but the gist was that "gmail.com" is a configuration value that can be changed at runtime, and someone changed the configuration. Thus, *@gmail.com stopped being a valid address, and they returned "that mailbox is unavailable".
If you don't want to scroll to the other thread, here's the postmortem: https://static.googleusercontent.com/media/www.google.com/en...
Re: Post Mortem of Google Outage on 14 December 2020
#19TLDR: the team forgot to update the resource quota requirements for a critical component of Google’s authentication system while transitioning between quota systems.
Is that an accurate tl;dr?
"As part of an ongoing migration of the User ID Service to a new quota system, a change was made in October to register the User ID Service with the new quota system, but parts of the previous quota system were left in place which incorrectly reported the usage for the User ID Service as 0."
Re: Post Mortem of Google Outage on 14 December 2020
#20off-topic :Do we know what happened a day after that when Gmail returned "this email doesn't exist" ?
Here is the incident report for the Gmail problem: https://static.googleusercontent.com/media/www.google.com/en... It is linked from the Google Workspace status page here: https://www.google.com/appsstatus#hl=en-GB&v=issue&sid=1&iid...
thanks, so a change of an env variable put the most used email system in the world down during 6 hours, not sure I can believe that