Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Post Mortem of Google Outage on 14 December 2020

status.cloud.google.com

Post reply on HN
Print view View original

41–50 of 209 posts

Re: Post Mortem of Google Outage on 14 December 2020

#41

Post by pdkl95 »

azornathogron wrote: post #5
polote wrote: post #2

off-topic :Do we know what happened a day after that when Gmail returned "this email doesn't exist" ?

Here is the incident report for the Gmail problem: https://static.googleusercontent.com/media/www.google.com/en... It is linked from the Google Workspace status page here: https://www.google.com/appsstatus#hl=en-GB&v=issue&sid=1&iid...

> A configuration change during this migration shifted the formatting behavior of a service option so that it incorrectly provided an invalid domain name, instead of the intended "gmail.com" domain name, to the Google SMTP inbound service.

Wow... how was this even possible? Did they do any testing whatsoever before migrating the live production system? They misformatting the domain name should have broken even basic functionality tests.

I wonder if they didn't actually test the literal "gmail.com" configuration, due to dev/testing environments using a different domain name? I had that problem when on my first Ruby on Rails project due to subtle differences between the development/test/production settings in config/environments/. Running "rake test" is not a substitute for an actual test of the real production system.

Re: Post Mortem of Google Outage on 14 December 2020

#44

Post by inopinatus »

physicsgraph wrote: post #11

My favorite quote: "prevent fast implementation of global changes." This is how large organizations become slower compared to small "nimble" companies. Everyone fails, but the noticeability of failure incentivizes large organizations to be more careful.

It doesn’t have to be this way, but that’s partly a matter of culture. By aspiring to present/think/act as a monoplatform, Google risks substantially increasing the blast radius of individual component failure. A global quota system mediating every other service sounds both totally on brand, and also the antithesis of everything I learned about public cloud scaling at AWS. There we made jokes, that weren’t jokes, about service teams essentially DoS’ing each other, and this being the natural order of things that every service must simply be resilient to and scale for.

Having been impressed upon by that mindset, my design reflex is instead to aim for elimination of global dependencies entirely, rather than globally rate-limiting the impact of a global rate-limiter.

I’m not saying either is a right answer, but that there are consequences to being true to your philosophy. There are upsides, too, with Google’s integrated approach, notable particularly when you build end-to-end systems from public cloud service portfolios and benefit from consistency in product design, something AWS eschews in favour of sometimes radical diversity. I see these emergent properties of each as an inevitability, a kind of generalised Conway’s Law.

Re: Post Mortem of Google Outage on 14 December 2020

#45

Post by cpncrunch »

tpolm wrote: post #36

it is interesting that is both cases (recent gmail and this one) it was a "migration": "As part of an ongoing migration of the User ID Service to a new quota system" "An ongoing migration was in effect to update this underlying configuration system" it was not a new feature, not a massive hardware failure, it was migrating part of the working product due to some unclear reason of "best practices". both of those migra…

>both of those migrations failed with symptoms suggesting that whoever was performing them did not have deep understanding of systems architecture or safety practices and there was no one to stop them from failing.

Can any single person at Google have a full understanding of all the dependencies for even a single system? I have no idea, as I've never worked there, but I would imagine that there is a lot of complexity.

Re: Post Mortem of Google Outage on 14 December 2020

#46

Post by gojomo »

b0afc375b5 wrote: post #42
sabujp wrote: post #27

s/post mortem/incident analysis/

What is the difference between the two? I tried searching for 'post mortem vs incident analysis' but couldn't find anything.

Some org might have decreed internal fine distinctions, but in common parlance, both terms are used for the same kind of after-event writeups.

Re: Post Mortem of Google Outage on 14 December 2020

#47

Post by ph0rque »

b0afc375b5 wrote: post #42
sabujp wrote: post #27

s/post mortem/incident analysis/

What is the difference between the two? I tried searching for 'post mortem vs incident analysis' but couldn't find anything.

Well, post mortem means "after death" in Latin. So it would seem the difference is, one can recover from an incident...

Re: Post Mortem of Google Outage on 14 December 2020

#49

Post by x87678r »

I'm curious about big outages like this in big internet corps.

Does anyone know if SREs in Europe fixed the problem or it relied on people in Mountain View? When an outage this big hits do devs get involved?

I've worked as an SRE and it sucks to be fixing developer's bugs in the middle of the night.

Re: Post Mortem of Google Outage on 14 December 2020

#50

Post by papito »

ak217 wrote: post #32

I've worked on a number of systems where code was pushed to a staging environment (a persistent functional replica of production where integration tests happen) and sat there for a week before being allowed in production. A staging setup might have prevented this scenario, since the quota enforcement grace period would expire in staging a week before prod and give the team a week to notice and push an emergency fix.

Staging is never hammered with production traffic, often not exposing problems. It's a sanity check for developers and QA, essentially.

On the scale of Google, you test in production, but with careful staged canary deployments. Even a 1% rollout is more than most of us have ever dealt with.

Post reply on HN