Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Post Mortem of Google Outage on 14 December 2020

status.cloud.google.com

Post reply on HN
Print view View original

141–150 of 209 posts

Re: Post Mortem of Google Outage on 14 December 2020

#141

Post by hashhar »

3np wrote: post #129
jiggawatts wrote: post #63

I've been bitten by quota grace periods before, they're the "buffer bloat" of cloud platform management systems. You build something, it seems fine, and then a month later it keels over. My tiny disaster was caused by Amazon EFS. They provide a performance quota grace period of a month, during which all tests passed with flying colours. Turns out that EFS rations out IOPS per GB of stored data, and this particular ap…

My unexpected AWS quota story: AWS Batch. Using managed instances, just asking AWS to provision “suitable and available”. Daily jobs (docker containers) ran fine for the first 6 months. Eventually started stalling and failing with DockerTimeout. They would fail in batches of 100s and then at some point start working again. It took several hours and some back and forth with support to realize that the burst IOPS quota…

I've been bit by the same issue on ECS.

Some stream processing applications restarted and they perform a lot of reads at startup to recover their in-memory state. All other containers on that instance eventually also restarted and got migrated to another EC2 instance which also got IOPS depleted soon enough.

And the cycle continues. The issue was there was no proper monitoring set up and getting a timeout from Docker isn't very helpful error message.

Since then I've made sure to build in checks to prevent bouncing all services on a machine at once and spreading out applications that use disk across machines instead of binpacking.

Re: Post Mortem of Google Outage on 14 December 2020

#142

Post by gregoda »

ph0rque wrote: post #47
b0afc375b5 wrote: post #42

Earlier quoted context omitted.

What is the difference between the two? I tried searching for 'post mortem vs incident analysis' but couldn't find anything.

Well, post mortem means "after death" in Latin. So it would seem the difference is, one can recover from an incident...

I'd tend to think that 'post-mortem' translates (in common usage) more accurately as 'after termination' -- a process performed at the completion of some other process. It's really a good idea to do post-mortems on successes as well as failures.

Re: Post Mortem of Google Outage on 14 December 2020

#143

Post by jimmydorry »

growse wrote: post #112
b112 wrote: post #104

Earlier quoted context omitted.

Email is extremely non-brittle, and everyone following the RFC is one main way it is able to be non-brittle. What is naive here, is that you think things are the way there are, for no reason. Mail is decades old, built upon millions of hours of work crafting software, RFC standards, and works the way it does, including bounces, to ensure stability. A 5xx error means 'perm failure'. There are a variety of 5xx class re…

> Email is extremely non-brittle, and everyone following the RFC is one main way it is able to be non-brittle. What is naive here, is that you think things are the way there are, for no reason. > Mail is decades old, built upon millions of hours of work crafting software, RFC standards, and works the way it does, including bounces, to ensure stability. Same people argue that email is a hot mess exactly because it's d…

If you hosted a mail server, would you really want an ever increasing number of mail servers hitting your mail server for invalid email addressess for an indeterminate period of time? The 5xx error code you are returning can't be trusted, after all.

That seems to be what you are advocating for.

Re: Post Mortem of Google Outage on 14 December 2020

#144

Post by 3np »

hashhar wrote: post #141
3np wrote: post #129

Earlier quoted context omitted.

My unexpected AWS quota story: AWS Batch. Using managed instances, just asking AWS to provision “suitable and available”. Daily jobs (docker containers) ran fine for the first 6 months. Eventually started stalling and failing with DockerTimeout. They would fail in batches of 100s and then at some point start working again. It took several hours and some back and forth with support to realize that the burst IOPS quota…

I've been bit by the same issue on ECS. Some stream processing applications restarted and they perform a lot of reads at startup to recover their in-memory state. All other containers on that instance eventually also restarted and got migrated to another EC2 instance which also got IOPS depleted soon enough. And the cycle continues. The issue was there was no proper monitoring set up and getting a timeout from Docker…

The icing on the cake here is that those IOPS from the docker agent is outside of your control. Before having to dive deeper into it, I would have assumed that only IOPS stemming from the workloads themselves would count against quotas.

Using these abstractions of abstractions of abstractions that all end up leaking fatal failure modes you have to deal with yourself makes me start questioning the fundamental value proposal. The one major thing you get away from is setup costs, but the total time investment gets amortized.

Re: Post Mortem of Google Outage on 14 December 2020

#145

Post by byecomputer »

speedgoose wrote: post #135
maxander wrote: post #53

Earlier quoted context omitted.

The odds that they would happen simultaneously if they’re completely unrelated seem astronomically small, certainly? Both are noted as being related to “ongoing migrations,” though AFAICT not related ones. I would bet there’s a human factor connection- e.g., the day before there was a big meeting where a higher-level management gave multiple ops teams go-ahead on their respective plans, resulting in a multiple potent…

I think the likelyhoood of two incidents happening in the same period is not astronomically small, and is a variant of the birthday problem. It's a bit counter intuitive but if you have a few incidents during a year, the probability to have two incidents the same week is a lot higher than what you would expect. https://en.m.wikipedia.org/wiki/Birthday_problem

The birthday problem involves random people with unrelated birthdays. This is like two not-random siblings both calling in sick in the same week. The case of two large Google outrages where the whole service gets conked has much more potential to have a shared or related cause than two birthdays happening at once. You're right that it's not astronomically small, but the odds of them being related seem healthier than the odds of them being unrelated.

Re: Post Mortem of Google Outage on 14 December 2020

#146

Post by csnweb »

3np wrote: post #129
jiggawatts wrote: post #63

I've been bitten by quota grace periods before, they're the "buffer bloat" of cloud platform management systems. You build something, it seems fine, and then a month later it keels over. My tiny disaster was caused by Amazon EFS. They provide a performance quota grace period of a month, during which all tests passed with flying colours. Turns out that EFS rations out IOPS per GB of stored data, and this particular ap…

My unexpected AWS quota story: AWS Batch. Using managed instances, just asking AWS to provision “suitable and available”. Daily jobs (docker containers) ran fine for the first 6 months. Eventually started stalling and failing with DockerTimeout. They would fail in batches of 100s and then at some point start working again. It took several hours and some back and forth with support to realize that the burst IOPS quota…

All these weird quotas and complicated pricing were the reason why we switched to digitalocean once they had managed kubernetes. It's such a pleasure to not have to worry about it and also much cheaper.

Re: Post Mortem of Google Outage on 14 December 2020

#147

Post by b112 »

growse wrote: post #112
b112 wrote: post #104

Earlier quoted context omitted.

Email is extremely non-brittle, and everyone following the RFC is one main way it is able to be non-brittle. What is naive here, is that you think things are the way there are, for no reason. Mail is decades old, built upon millions of hours of work crafting software, RFC standards, and works the way it does, including bounces, to ensure stability. A 5xx error means 'perm failure'. There are a variety of 5xx class re…

> Email is extremely non-brittle, and everyone following the RFC is one main way it is able to be non-brittle. What is naive here, is that you think things are the way there are, for no reason. > Mail is decades old, built upon millions of hours of work crafting software, RFC standards, and works the way it does, including bounces, to ensure stability. Same people argue that email is a hot mess exactly because it's d…

Same people argue that email is a hot mess

You're lumping together "email content" with "SMTP". SMTP isn't a "patchwork of standards", it has a very specific RFC for it. SMTP works very, very, very well.

Configuration issues can be, and often are, temporary. Also, bugs are a thing.

This isn't just "a bug" or "a config issue", this is an edge case bug or config issue. I've handled literally hundreds of thousands of mail servers in high avail production, with some of those being extremely high volume.

What you're wanting to change the normal flow of operation, for an extremely rare edge case.

Of course there's a semantic difference between "come back later" and "go away forever", I'm just arguing that a client that doesn't blindly trust what every connected system tells it is going to more often successfully achieve its goals than one which does.

Not true here. In 99.999999% of cases when you get a 5xx response code, the correct thing to do is immediate bounce. This sort of mess, 5xx 'by accident', is insanely rare.

And it's not 'go away forever', it's "this specific mail cannot be delivered, please return it to sender, so they can examine the issue and deal with it".

It's "bring a human into the equation".

How is this an issue? For something which is very, very rare.

Re: Post Mortem of Google Outage on 14 December 2020

#148

Post by aleph1 »

herf wrote: post #95

In Thunderbird, OAuth2 login is still broken. The login page prompts for email again and again, never makes it to the password.

This could be caused by Google not recognizing and blocking Thunderbird's default user agent; try toggling general.useragent.compatMode.firefox to true (which basically has TB emulate Firefox's user agent)

Re: Post Mortem of Google Outage on 14 December 2020

#149

Post by ryanobjc »

x87678r wrote: post #49

I'm curious about big outages like this in big internet corps. Does anyone know if SREs in Europe fixed the problem or it relied on people in Mountain View? When an outage this big hits do devs get involved? I've worked as an SRE and it sucks to be fixing developer's bugs in the middle of the night.

4am is noon Europe time, so the sres in Europe would have gotten the page and been on top of their game. They fixed it pretty quick.

Of course in a global outage, nothing is fast enough.

Re: Post Mortem of Google Outage on 14 December 2020

#150

Post by b112 »

jiggawatts wrote: post #127
jhgg wrote: post #110

Earlier quoted context omitted.

To be fair, there is a cloud watch metric you could have set an alert on: "PercentIOLimit"

Ahhh... the joy of enterprise monitoring systems that do exactly nothing by default, and are very helpful in avoiding any further recurrences of one-time issues. At a nominal fee, of course. The golden rule of both backups and monitoring is: There are no time machines. It's not helpful to find out after the fact that a default-off alert or metric threshold alarm could have avoided the issue. It's not helpful to blame…

I'll be the fly in the ointment here, the additional sand in your shorts, and say "This is why bare metal is better".

Bare metal doesn't have to be VM/containerless. Roll your own. But at least in this case, you're dealing with your own issues, with things not hidden and abstracted away, and at literally 1/100th to 1/10000th the cost of AWS.

And yes, that's with the hardware investment and wage costs rolled in.

I feel like AWS and others created some sort of one ring, and just reeled admins in, pulling them from the wild, so that with a shortage of greybeards, people can't find those with the 'sysadmin temperament', and are therefore stuck with cloud.

And it is indeed a specific temperament, to create stability, which means creating some constraint, yet to forge that constraint in the most effective way, whilst doing one's best to enable devs to work most effectively.

AWS has constraints, but because it's a wall of "this is the way external corp does it", it gets far less flack than the guy that smells like onions, when he says "we can't do this safely" or "this policy must be followed".

Yes, OK, this is a bit of a rant. Sorry.

Post reply on HN