s/post mortem/incident analysis/
What is the difference between the two? I tried searching for 'post mortem vs incident analysis' but couldn't find anything.
Post-mortem is just a gruesome term.
Post Mortem of Google Outage on 14 December 2020
51–60 of 209 posts
Re: Post Mortem of Google Outage on 14 December 2020
#52Earlier quoted context omitted.
thanks, so a change of an env variable put the most used email system in the world down during 6 hours, not sure I can believe that
>> so a change of an env variable put the most used email system in the world down during 6 hours, not sure I can believe that I can totally believe it. In my experience, the bigger the outage the stupider-seeming the cause.
Google has had a bunch of notorious outages caused by similar things, including pushing a completely blank front-end load balancer config to global production. The post mortem action items for these are always really deep thoughts about the safety of config changes but in my experience there, nobody ever really fixes them because the problem is really hard.
For this kind of change I would probably have wanted some kind of shadow system that loaded the new config, received production inputs, produced responses that were monitored but discarded, and had no other observable side effects. That's such a pain in the ass that most teams aren't going to bother setting that up, even when the risks are obvious.
Re: Post Mortem of Google Outage on 14 December 2020
#53Earlier quoted context omitted.
Different outage. The Gmail postmortem is linked in another thread, but the gist was that "gmail.com" is a configuration value that can be changed at runtime, and someone changed the configuration. Thus, *@gmail.com stopped being a valid address, and they returned "that mailbox is unavailable". If you don't want to scroll to the other thread, here's the postmortem: https://static.googleusercontent.com/media/www.googl…
Indeed, it seems to be a completely unrelated issue. Two major Google outages on the same day for different reasons.
The odds that they would happen simultaneously if they’re completely unrelated seem astronomically small, certainly?
Both are noted as being related to “ongoing migrations,” though AFAICT not related ones. I would bet there’s a human factor connection- e.g., the day before there was a big meeting where a higher-level management gave multiple ops teams go-ahead on their respective plans, resulting in a multiple potentially breaking changes occurring at the same time.
Re: Post Mortem of Google Outage on 14 December 2020
#54Thought: There should be an official Google status dashboard for "free" (paid for via personal ad targeting metadata) services like Search, Gmail, Drive, etc. With postmortems, too. Probably won't happen unless it's mandated by law, though.
Keep your scare quotes.
Re: Post Mortem of Google Outage on 14 December 2020
#55Earlier quoted context omitted.
Here is the incident report for the Gmail problem: https://static.googleusercontent.com/media/www.google.com/en... It is linked from the Google Workspace status page here: https://www.google.com/appsstatus#hl=en-GB&v=issue&sid=1&iid...
That Google is choosing to use a PDF (!) as the official incident-reporting media is as confidence-destroying as was the outage.
It's almost as disappointing as the fact that their status page doesn't redirect from HTTP to HTTPS.
(Presumably they wanted to make the status page depend on as few services as possible, to prevent a scenario where an outage also affects the status page itself, but whatever script they are using to publish updates to the page could also perform a check that the HTTPS version of the site is accessible, and if not, remove the redirect).
Could we get the URL of the submission updated please? (Also, it would be nice if the submission form added an "Are you sure?" step when people submit HTTP links).
Re: Post Mortem of Google Outage on 14 December 2020
#56it is interesting that is both cases (recent gmail and this one) it was a "migration": "As part of an ongoing migration of the User ID Service to a new quota system" "An ongoing migration was in effect to update this underlying configuration system" it was not a new feature, not a massive hardware failure, it was migrating part of the working product due to some unclear reason of "best practices". both of those migra…
>both of those migrations failed with symptoms suggesting that whoever was performing them did not have deep understanding of systems architecture or safety practices and there was no one to stop them from failing. Can any single person at Google have a full understanding of all the dependencies for even a single system? I have no idea, as I've never worked there, but I would imagine that there is a lot of complexity…
somehow they managed to build complex systems like gmail, continuously develop new features there and not have massive outages due to "migrations" - suggests that something that they were doing right, they are no longer able to do
Re: Post Mortem of Google Outage on 14 December 2020
#57My favorite quote: "prevent fast implementation of global changes." This is how large organizations become slower compared to small "nimble" companies. Everyone fails, but the noticeability of failure incentivizes large organizations to be more careful.
Slow rollouts can be a double-edged sword, too:
> a change was made in October to register the User ID Service with the new quota system, but parts of the previous quota system were left in place which incorrectly reported the usage for the User ID Service as 0. An existing grace period on enforcing quota restrictions delayed the impact, which eventually expired, triggering automated quota systems to decrease the quota allowed for the User ID service and triggering this incident.
Grace period on enforcement of a major policy change is an excellent practice...but it also means months can go by between the introduction of a problem and when the problem actually surfaces. That can lead to increased time-to-resolution because many engineers won't have that months-old change at the front of their mind while debugging.
Re: Post Mortem of Google Outage on 14 December 2020
#58I'm curious about big outages like this in big internet corps. Does anyone know if SREs in Europe fixed the problem or it relied on people in Mountain View? When an outage this big hits do devs get involved? I've worked as an SRE and it sucks to be fixing developer's bugs in the middle of the night.
Yes, SRE teams typically have a sister SRE team in another continent and time zone.
Re: Post Mortem of Google Outage on 14 December 2020
#59Earlier quoted context omitted.
>> so a change of an env variable put the most used email system in the world down during 6 hours, not sure I can believe that I can totally believe it. In my experience, the bigger the outage the stupider-seeming the cause.
Google has had a bunch of notorious outages caused by similar things, including pushing a completely blank front-end load balancer config to global production. The post mortem action items for these are always really deep thoughts about the safety of config changes but in my experience there, nobody ever really fixes them because the problem is really hard. For this kind of change I would probably have wanted some ki…
Actually now that I remember correctly, back when I was in that barber shop quartet in Skokie^W^W^W^W^W err, back when I was an SRE on Gmail's delivery subsystem, we actually did recognize the incredible risk posed by the delivery config system and our team developed what was known as "the finch", a tiny production shard that loaded the config before all others. It was called the finch to distinguish it from "the canary" which was generally used for deploying new builds. I wonder if these newfangled "best practices" threw the finch under the bus.
Re: Post Mortem of Google Outage on 14 December 2020
#60Hmm one thing that jumped out at me was the organizational mistake of having a very long automated "grace period". This is actually bad system architecture. Whenever you have a timeout for something that involves a major config change like this, the timeout must be short (like less than a week). Otherwise, it is very likely people will forget about it, and it will take a while for people to recognize and fix the prob…
I agree, and even if the grace period were a good idea, enforcement should have slowly ratcheted up over the grace period, rather than having full enforcement immediately after it expired.