Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Post Mortem of Google Outage on 14 December 2020

status.cloud.google.com

Post reply on HN
Print view View original

21–30 of 209 posts

Re: Post Mortem of Google Outage on 14 December 2020

#21

Post by mehrdadn »

klysm wrote: post #10
mehrdadn wrote: post #9

I'm skimming this but so far I still don't understand how any kind of authentication failure can or should lead to an SMTP server returning "this address doesn't exist". Does anyone see an explanation? Edit 1: Oh wow I didn't even realize there were multiple incidents. Thanks! Edit 2 (after reading the Gmail post-mortem): AH! It was a messed up domain name! When this event happened, I said senders need to avoid takin…

If I had to guess I would say sometimes when you request something you aren't authorized to see you get a 404 because they don't want you to be able to tell what exists or not without any creds.

Sending an email doesn't require "seeing" anything other than whether the server is willing to receive email for that address though, right?

Also, that problem occurs due to lack of authorization, not due to authentication failure, right? Authentication failure is a different kind of error to the client—if the server fails to authenticate you, clearly you already know that it's not going to show you anything?

Re: Post Mortem of Google Outage on 14 December 2020

#22

Post by jeffbee »

azornathogron wrote: post #5
polote wrote: post #2

off-topic :Do we know what happened a day after that when Gmail returned "this email doesn't exist" ?

Here is the incident report for the Gmail problem: https://static.googleusercontent.com/media/www.google.com/en... It is linked from the Google Workspace status page here: https://www.google.com/appsstatus#hl=en-GB&v=issue&sid=1&iid...

Damn, that's awful. Should lead to a deep questioning of people who claim to be migrating your junk to "best practices" if the old config system worked fine for 15 years and the new one caused a massive dataloss outage on Day 1.

Re: Post Mortem of Google Outage on 14 December 2020

#24

Post by adrianmonk »

jeffbee wrote: post #13
mehrdadn wrote: post #9

I'm skimming this but so far I still don't understand how any kind of authentication failure can or should lead to an SMTP server returning "this address doesn't exist". Does anyone see an explanation? Edit 1: Oh wow I didn't even realize there were multiple incidents. Thanks! Edit 2 (after reading the Gmail post-mortem): AH! It was a messed up domain name! When this event happened, I said senders need to avoid takin…

That's a different outage. SMTP 550 was the next day.

Some outage information showing the SMTP 550 incident was at a different time:

"Gmail - Service Details" ( https://www.google.com/appsstatus#hl=en&v=issue&sid=1&iid=a8...) shows 12/15/20 as the date.

That links to "Google Cloud Issue Summary" "Gmail - 2020-12-14 and 2020-12-15" (https://static.googleusercontent.com/media/www.google.com/en...) which mentions the 550 error code.

Re: Post Mortem of Google Outage on 14 December 2020

#25

Post by Xorlev »

physicsgraph wrote: post #11

My favorite quote: "prevent fast implementation of global changes." This is how large organizations become slower compared to small "nimble" companies. Everyone fails, but the noticeability of failure incentivizes large organizations to be more careful.

Slowing down actuation of prod changes to be over hours vs. seconds is a far cry from the large org / small org problem. Ultimately, when the world depends on you, limiting the blast radius to X% of the world vs. 100% of it is a significant improvement.

Re: Post Mortem of Google Outage on 14 December 2020

#26

Post by [deleted] »

jrockway wrote: post #18
mehrdadn wrote: post #9

I'm skimming this but so far I still don't understand how any kind of authentication failure can or should lead to an SMTP server returning "this address doesn't exist". Does anyone see an explanation? Edit 1: Oh wow I didn't even realize there were multiple incidents. Thanks! Edit 2 (after reading the Gmail post-mortem): AH! It was a messed up domain name! When this event happened, I said senders need to avoid takin…

Different outage. The Gmail postmortem is linked in another thread, but the gist was that "gmail.com" is a configuration value that can be changed at runtime, and someone changed the configuration. Thus, *@gmail.com stopped being a valid address, and they returned "that mailbox is unavailable". If you don't want to scroll to the other thread, here's the postmortem: https://static.googleusercontent.com/media/www.googl…

[deleted]

Re: Post Mortem of Google Outage on 14 December 2020

#28

Post by Splendor »

tpmx wrote: post #3

Thought: There should be an official Google status dashboard for "free" (paid for via personal ad targeting metadata) services like Search, Gmail, Drive, etc. With postmortems, too. Probably won't happen unless it's mandated by law, though.

There are several. Here's one example: https://downdetector.com/

Re: Post Mortem of Google Outage on 14 December 2020

#30

Post by [deleted] »

jrockway wrote: post #18
mehrdadn wrote: post #9

I'm skimming this but so far I still don't understand how any kind of authentication failure can or should lead to an SMTP server returning "this address doesn't exist". Does anyone see an explanation? Edit 1: Oh wow I didn't even realize there were multiple incidents. Thanks! Edit 2 (after reading the Gmail post-mortem): AH! It was a messed up domain name! When this event happened, I said senders need to avoid takin…

Different outage. The Gmail postmortem is linked in another thread, but the gist was that "gmail.com" is a configuration value that can be changed at runtime, and someone changed the configuration. Thus, *@gmail.com stopped being a valid address, and they returned "that mailbox is unavailable". If you don't want to scroll to the other thread, here's the postmortem: https://static.googleusercontent.com/media/www.googl…

[deleted]
Post reply on HN