Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Post Mortem of Google Outage on 14 December 2020

status.cloud.google.com

Post reply on HN
Print view View original

151–160 of 209 posts

Re: Post Mortem of Google Outage on 14 December 2020

#151

Post by diveanon »

csnweb wrote: post #146
3np wrote: post #129

Earlier quoted context omitted.

My unexpected AWS quota story: AWS Batch. Using managed instances, just asking AWS to provision “suitable and available”. Daily jobs (docker containers) ran fine for the first 6 months. Eventually started stalling and failing with DockerTimeout. They would fail in batches of 100s and then at some point start working again. It took several hours and some back and forth with support to realize that the burst IOPS quota…

All these weird quotas and complicated pricing were the reason why we switched to digitalocean once they had managed kubernetes. It's such a pleasure to not have to worry about it and also much cheaper.

Even more so when you factor in dev time.

Re: Post Mortem of Google Outage on 14 December 2020

#152

Post by ryanobjc »

cpncrunch wrote: post #45
tpolm wrote: post #36

it is interesting that is both cases (recent gmail and this one) it was a "migration": "As part of an ongoing migration of the User ID Service to a new quota system" "An ongoing migration was in effect to update this underlying configuration system" it was not a new feature, not a massive hardware failure, it was migrating part of the working product due to some unclear reason of "best practices". both of those migra…

>both of those migrations failed with symptoms suggesting that whoever was performing them did not have deep understanding of systems architecture or safety practices and there was no one to stop them from failing. Can any single person at Google have a full understanding of all the dependencies for even a single system? I have no idea, as I've never worked there, but I would imagine that there is a lot of complexity…

To answer your question, the answer is yes. Some people do understand the dep stack. Takes years but hey there are lifers.

Re: Post Mortem of Google Outage on 14 December 2020

#153

Post by pmlnr »

throwaway201103 wrote: post #96
growse wrote: post #90

Earlier quoted context omitted.

Generally speaking, if you're trying to write a reliable distributed system (and email is a massively distributed system), a good principle to follow is to retry on failure, no matter the failure. Obviously there are edge cases, and obviously you don't just retry every 1ms forever, but to assume that an error that comes back from a system you've called once is both (a) the product of a 100% working non-faulty system…

This makes email delivery much more complicated than it already is, for no good reason. There are no guarantees when it comes to email delivery anyway.

But there are. At least there supposed to be: a 2xx should be issued when the mail is written to storage, and not before.

Re: Post Mortem of Google Outage on 14 December 2020

#154

Post by byecomputer »

shadowgovt wrote: post #132
tpolm wrote: post #36

it is interesting that is both cases (recent gmail and this one) it was a "migration": "As part of an ongoing migration of the User ID Service to a new quota system" "An ongoing migration was in effect to update this underlying configuration system" it was not a new feature, not a massive hardware failure, it was migrating part of the working product due to some unclear reason of "best practices". both of those migra…

I think you're on to something. One of the challenges that teams at Google have is service dependencies. In theory, Google is one big, happy family and everyone is responsible for everyone's code. In practice, teams have focuses, software interdepends and interoperates, and mistakes get made at the margin where the linkages between two software systems are neither team's direct responsibility---or the responsibility…

Really, universal problems that all large corporations and bureaucracies are vulnerable to. Overlaps or gaps in responsibility (breeding plausible deniability), lack of communication, and management issues (hence dysfunctional incentives) are difficult to root out and cost a lot of money to track down and fix.

Re: Post Mortem of Google Outage on 14 December 2020

#155

Post by mtsr »

jiggawatts wrote: post #63

I've been bitten by quota grace periods before, they're the "buffer bloat" of cloud platform management systems. You build something, it seems fine, and then a month later it keels over. My tiny disaster was caused by Amazon EFS. They provide a performance quota grace period of a month, during which all tests passed with flying colours. Turns out that EFS rations out IOPS per GB of stored data, and this particular ap…

And this is why it’s not easy to use any of the cloud platforms parttime. They’re all full of these kinds of gotchas and your example is even a minor one since it really is quite clear from the documentation. There’s even a choice of Provisioned Throughput if you’re clicking this together in the Console, which should probably make you suspicious about what happens if you don’t use that.

But a certified AWS Architect, especially at the higher levels will easily spot the big gotchas from just your architecture diagram. Consider getting certified yourself, if you want to be more confident around these in the future. But there really is no replacement for just putting in a lot of hours.

Re: Post Mortem of Google Outage on 14 December 2020

#156

Post by waheoo »

jiggawatts wrote: post #63

I've been bitten by quota grace periods before, they're the "buffer bloat" of cloud platform management systems. You build something, it seems fine, and then a month later it keels over. My tiny disaster was caused by Amazon EFS. They provide a performance quota grace period of a month, during which all tests passed with flying colours. Turns out that EFS rations out IOPS per GB of stored data, and this particular ap…

I've seen this same iops story play out 2 different times at two different companies both with the same story on production systems.

Both were solved by the same thing,

Provisioning a few terabytes of storage space with nothing on it.

Re: Post Mortem of Google Outage on 14 December 2020

#157

Post by bostonsre »

mtsr wrote: post #155
jiggawatts wrote: post #63

I've been bitten by quota grace periods before, they're the "buffer bloat" of cloud platform management systems. You build something, it seems fine, and then a month later it keels over. My tiny disaster was caused by Amazon EFS. They provide a performance quota grace period of a month, during which all tests passed with flying colours. Turns out that EFS rations out IOPS per GB of stored data, and this particular ap…

And this is why it’s not easy to use any of the cloud platforms parttime. They’re all full of these kinds of gotchas and your example is even a minor one since it really is quite clear from the documentation. There’s even a choice of Provisioned Throughput if you’re clicking this together in the Console, which should probably make you suspicious about what happens if you don’t use that. But a certified AWS Architect,…

Does the certification cover all products? They seem to be releasing products more and more frequently.. it has to be tough to keep up to speed on everything to a degree where you don't need Google to explore options and plan projects.

Re: Post Mortem of Google Outage on 14 December 2020

#158

Post by perlgeek »

tpolm wrote: post #36

it is interesting that is both cases (recent gmail and this one) it was a "migration": "As part of an ongoing migration of the User ID Service to a new quota system" "An ongoing migration was in effect to update this underlying configuration system" it was not a new feature, not a massive hardware failure, it was migrating part of the working product due to some unclear reason of "best practices". both of those migra…

> it was migrating part of the working product due to some unclear reason of "best practices".

I want to push back on that. Of course, the reasons are unclear to an outsider.

Migrations are an unavoidable in any system that is still evolving (i.e. not dead). Old designs turn out to be too limited or too slow for an evolved use case, so you migrate them to a new service or a new data structure.

If you try to avoid migrations by building The Perfect Things[tm] upfront, you get lost in overengineering instead.

In my own work, I do migrations with some regularity, and they all have a clear goal, it's never what you call 'some unclear reason of "best practices"'.

Re: Post Mortem of Google Outage on 14 December 2020

#159

Post by CodesInChaos »

rckoepke wrote: post #113
vlovich123 wrote: post #77

Earlier quoted context omitted.

It’s not because the underlying storage is 1 TiB disk, so you’re getting a fixed allocation that’s shared among many other 50mb/s clients (ie assuming flash and overall transfer rates of ~2gigabytes/, that lets them coex roughly 40 customers on 1 machine without over subscribing)? Isn’t that kind of pricing mode about the only one that would be feasible to implement to make this cost effective? How are you thinking i…

The pedantic answer would be to allow customers to specify both minimum storage space and minimum IOPS. Then charge them for whichever is the larger portion of a drive and provide the space and IOPS that they are then paying for. This way, if you need 10MB but also 25,000 IOPS, then you'll pay for ~10% of a drive. You'll get your minimum required 25,000 IOPS....and also 200GB or whatever share of the drive is require…

For Amazon EBS you have some kind of IOPS slider. This is possible even on normal EBS (gp3) volumes, not just on those special high performance EBS volumes (io2).

> General Purpose SSD (gp3) - IOPS 3,000 IOPS free and $0.006/provisioned IOPS-month over 3,000

> General Purpose SSD (gp3) - Throughput 125 MB/s free and $0.0476/provisioned MB/s-month over 125

https://aws.amazon.com/ebs/pricing/

Re: Post Mortem of Google Outage on 14 December 2020

#160

Post by TedShiller »

To be honest, I never understood the point of companies publishing post mortems after outages. What is it supposed to accomplish? We the users don’t understand their infrastructure anyway, how are we expected to understand the post mortem? Besides, as the users why should we care why things went down? The fact is, they did. And that’s the only thing that matters. I don’t feel better knowing WHY I lost my services. I just feel bad BECAUSE I lost them. Or are post mortems supposed to reassure users that the outage won’t happen again? It doesn’t reassure that either because by definition unpredictable outages always happen due to something new and unpredictable. This post mortem certainly won’t stop the next outage from happening. We KNOW there will be more, we just don’t know when. Or are post mortems supposed to show that the company takes full responsibility for what happened? But they always do and are fully expected to. So it’s meaningless. No company would ever say “we don’t take responsibility for this error we caused”. Even in the case of massive data leaks, which cannot be reversed, companies always take full responsibility. And it doesn’t help anyone.

The only thing post mortems show is that the company didn’t do their job or was careless or disorganized or confused. But we already know that, because they had an outage.

So what’s the point?

Post reply on HN