Post Mortem of Google Outage on 14 December 2020

Post Mortem of Google Outage on 14 December 2020
status.cloud.google.com

https://static.googleusercontent.com/media/www.google.com/en...

It is linked from the Google Workspace status page here:

https://www.google.com/appsstatus#hl=en-GB&v=issue&sid=1&iid...

Edit 1: Oh wow I didn't even realize there were multiple incidents. Thanks!

Edit 2 (after reading the Gmail post-mortem): AH! It was a messed up domain name! When this event happened, I said senders need to avoid taking "invalid address" at face value when they've recently succeeded delivering to the same addresses. But despite the RFC saying senders "should not" repeat requests (rather than "must not"), many people had a lot of resistance to this idea, and instead just blamed Google for messing up implementing the RFC. People seemed to think every other server was right to treat this as a permanent error. But this post-mortem makes it crystal clear how that completely missed the point, and in fact wasn't the case at all. The part of the service concluding "address not found" was working correctly! They didn't mess up implementing the spec—they messed up their domain name in the configuration. Which is an operational error you just can't assume will never happen—it's exactly like the earlier analogy to someone opening the door to the mailman and not recognizing the name of a resident. A robust system (in this case the sender) needs to be able to recognize sudden anomalies—in this case, failure to deliver to an address that was accepting mail very recently—and retry things a bit later. You just can't assume nobody will make mistake operational mistakes, even if you somehow assume the software is bug-free.