Post Mortem of Google Outage on 14 December 2020 • Hacker News Board

Re: Post Mortem of Google Outage on 14 December 2020

#131

Post by shadowgovt » Sat, Dec 19, 2020, 4:55 AM UTC

ak217 wrote: post #32

I've worked on a number of systems where code was pushed to a staging environment (a persistent functional replica of production where integration tests happen) and sat there for a week before being allowed in production. A staging setup might have prevented this scenario, since the quota enforcement grace period would expire in staging a week before prod and give the team a week to notice and push an emergency fix.

In this specific case, the problem they were into is that the quota system had been different for three months, but the differences were not being enforced.

It's very unclear how they would have gone about Canary that period really, the canary should have been done on the quota side. The quota system should have enabled enforcement to 1% of the quota clients.

but it turns out it's actually hard to configure that sort of thing. The ways you can slice subsets of Google infrastructure are absolutely holographic, and it costs engineering time to change the slices.

Re: Post Mortem of Google Outage on 14 December 2020

#132

Post by shadowgovt » Sat, Dec 19, 2020, 5:00 AM UTC

tpolm wrote: post #36

it is interesting that is both cases (recent gmail and this one) it was a "migration": "As part of an ongoing migration of the User ID Service to a new quota system" "An ongoing migration was in effect to update this underlying configuration system" it was not a new feature, not a massive hardware failure, it was migrating part of the working product due to some unclear reason of "best practices". both of those migra…

I think you're on to something. One of the challenges that teams at Google have is service dependencies. In theory, Google is one big, happy family and everyone is responsible for everyone's code. In practice, teams have focuses, software interdepends and interoperates, and mistakes get made at the margin where the linkages between two software systems are neither team's direct responsibility---or the responsibility of both teams.

It's not malice, it's incentives and information flow. Integrating with a service that one is not responsible for, one can get tripped by unknown unknowns that the team that maintains the service has failed the document. And while a migration mistake is embarrassing, software engineering teams are generally rewarded for task completion, not for the time spent preparing for a failure that doesn't occur.

Re: Post Mortem of Google Outage on 14 December 2020

#133

Post by nullsense » Sat, Dec 19, 2020, 5:46 AM UTC

jiggawatts wrote: post #125

ineedasername wrote: post #108

Earlier quoted context omitted.

Even worse are undocumented quotas. Probably not quite the same thing, but I've had such an issue with a SaaS API for extracting data for a cloud system. It has an unpublished limit on queries. Writing a library of code to use it, I was fine. All tests passed. Running a full day's data pull & integration had sporadic failure: It would run w/o issue for half the job, then have seeming random errors on some requests. I…

There are some "fun" horror stories of people hitting the Azure Resource Manager API quota limits. At least one of them was linked to some sort of shared service principal for the client org, so if anyone hit it, the whole organisation could get locked out. It would first start to throw HTTP 429 codes occasionally, then you would get locked out for exponentially increasing times, up to two weeks or something absurd l…

Yikes. That sounds nightmarish.

Re: Post Mortem of Google Outage on 14 December 2020

#134

Post by caturopath » Sat, Dec 19, 2020, 5:48 AM UTC

wolco2 wrote: post #117

NoodleIncident wrote: post #61

Earlier quoted context omitted.

I think it's as simple as a case of the Mondays; you wouldn't roll out a migration like that on a Friday or the weekend, and rolling it out on a Monday gives you the least chances of problem occurring on those dates.

A Monday rollout sounds horrible. Probably the worst day for a rollout.

What about Saturday? Sunday? Friday?

Re: Post Mortem of Google Outage on 14 December 2020

#135

Post by speedgoose » Sat, Dec 19, 2020, 5:50 AM UTC

maxander wrote: post #53

cpncrunch wrote: post #39

Earlier quoted context omitted.

Indeed, it seems to be a completely unrelated issue. Two major Google outages on the same day for different reasons.

The odds that they would happen simultaneously if they’re completely unrelated seem astronomically small, certainly? Both are noted as being related to “ongoing migrations,” though AFAICT not related ones. I would bet there’s a human factor connection- e.g., the day before there was a big meeting where a higher-level management gave multiple ops teams go-ahead on their respective plans, resulting in a multiple potent…

I think the likelyhoood of two incidents happening in the same period is not astronomically small, and is a variant of the birthday problem. It's a bit counter intuitive but if you have a few incidents during a year, the probability to have two incidents the same week is a lot higher than what you would expect.

https://en.m.wikipedia.org/wiki/Birthday_problem

Re: Post Mortem of Google Outage on 14 December 2020

#136

Post by frankchn » Sat, Dec 19, 2020, 5:54 AM UTC

silentsea90 wrote: post #118

felixhuttmann wrote: post #116

Earlier quoted context omitted.

I often hear 'aim for elimination of global dependencies', but the reality is that there is no way around global dependencies. AWS STS or IAM is just as global as google's. The difference is that google more often builds with some form of guaranteed read-after-write consistency, while AWS is more often 'fail open'. For example, if you remove a permission from a user in GCP, you are guaranteed consistency within 7 min…

S3 is strongly consistent. https://aws.amazon.com/s3/consistency/ Which of Google's nosql db provides strong consistency - bigtable? Just confirming

S3 became strongly consistent only recently (https://aws.amazon.com/blogs/aws/amazon-s3-update-strong-rea...) while I think GCS and Azure Blob Storage has strong read-after-write consistency for a while now.

In any case, Cloud Spanner provides strong consistency in multi-region deployments.

Re: Post Mortem of Google Outage on 14 December 2020

#137

Post by caturopath » Sat, Dec 19, 2020, 6:07 AM UTC

tpolm wrote: post #56

cpncrunch wrote: post #45

Earlier quoted context omitted.

>both of those migrations failed with symptoms suggesting that whoever was performing them did not have deep understanding of systems architecture or safety practices and there was no one to stop them from failing. Can any single person at Google have a full understanding of all the dependencies for even a single system? I have no idea, as I've never worked there, but I would imagine that there is a lot of complexity…

somehow they managed to build complex systems like gmail, continuously develop new features there and not have massive outages due to "migrations" - suggests that something that they were doing right, they are no longer able to do

I'm pretty sure Google has had occasional severe outages for their whole history.

Re: Post Mortem of Google Outage on 14 December 2020

#138

Post by joshuamorton » Sat, Dec 19, 2020, 6:09 AM UTC

tpolm wrote: post #100

joshuamorton wrote: post #86

Earlier quoted context omitted.

A single event is not data.

there were more than two recent incidents lately: - YouTube outage this November 2020 - August 2020 outage of Google Suite including Gmail in both cases no postmortems were published

Afaik, Google doesn't publish public PMs for non-paid offerings, so youtube doesn't get a public pm.

For the August outage, I believe there was a public pm. That said I can't find it now (I think there was some link rot somewhere, and I've escalated about that).

Re: Post Mortem of Google Outage on 14 December 2020

#139

Post by gojomo » Sat, Dec 19, 2020, 6:16 AM UTC

gojomo wrote: post #38

azornathogron wrote: post #5

Earlier quoted context omitted.

Here is the incident report for the Gmail problem: https://static.googleusercontent.com/media/www.google.com/en... It is linked from the Google Workspace status page here: https://www.google.com/appsstatus#hl=en-GB&v=issue&sid=1&iid...

That Google is choosing to use a PDF (!) as the official incident-reporting media is as confidence-destroying as was the outage.

Wow, surprising PDF partisan downvotes here!

Re: Post Mortem of Google Outage on 14 December 2020

#140

Post by andreareina » Sat, Dec 19, 2020, 6:16 AM UTC

ineedasername wrote: post #108

jiggawatts wrote: post #63

I've been bitten by quota grace periods before, they're the "buffer bloat" of cloud platform management systems. You build something, it seems fine, and then a month later it keels over. My tiny disaster was caused by Amazon EFS. They provide a performance quota grace period of a month, during which all tests passed with flying colours. Turns out that EFS rations out IOPS per GB of stored data, and this particular ap…

Even worse are undocumented quotas. Probably not quite the same thing, but I've had such an issue with a SaaS API for extracting data for a cloud system. It has an unpublished limit on queries. Writing a library of code to use it, I was fine. All tests passed. Running a full day's data pull & integration had sporadic failure: It would run w/o issue for half the job, then have seeming random errors on some requests. I…

Google Maps^W Places API. Official limit is 100qps. Except that if your query has more than 20 results, the "next page" link doesn't work for between 1-2 seconds.