Live data from Hacker News

Post Mortem of Google Outage on 14 December 2020

status.cloud.google.com

111–120 of 209 posts

Re: Post Mortem of Google Outage on 14 December 2020

#111

I've been bitten by quota grace periods before, they're the "buffer bloat" of cloud platform management systems. You build something, it seems fine, and then a month later it keels over. My tiny disaster was caused by Amazon EFS. They provide a performance quota grace period of a month, during which all tests passed with flying colours. Turns out that EFS rations out IOPS per GB of stored data, and this particular ap…

I remember having something similar years ago. Ended up just creating a 100GB file, or something, just to get the IOPS I needed until we could migrate off EFS.

Re: Post Mortem of Google Outage on 14 December 2020

#112
post #104
post #90

Earlier quoted context omitted.

Generally speaking, if you're trying to write a reliable distributed system (and email is a massively distributed system), a good principle to follow is to retry on failure, no matter the failure. Obviously there are edge cases, and obviously you don't just retry every 1ms forever, but to assume that an error that comes back from a system you've called once is both (a) the product of a 100% working non-faulty system…

Email is extremely non-brittle, and everyone following the RFC is one main way it is able to be non-brittle. What is naive here, is that you think things are the way there are, for no reason. Mail is decades old, built upon millions of hours of work crafting software, RFC standards, and works the way it does, including bounces, to ensure stability. A 5xx error means 'perm failure'. There are a variety of 5xx class re…

> Email is extremely non-brittle, and everyone following the RFC is one main way it is able to be non-brittle. What is naive here, is that you think things are the way there are, for no reason.

> Mail is decades old, built upon millions of hours of work crafting software, RFC standards, and works the way it does, including bounces, to ensure stability.

Same people argue that email is a hot mess exactly because it's decades old and comprised of a patchwork of standards. It's not exactly the poster child of how to do this sort of thing well.

> For a mail system to respond with 5xx, when it is internally broken, is 100% a configuration issue. Every MTA on the planet, is designed (eg, postfix, sendmail, etc) to respond with a 4xx TEMP fail message if something is borked. A milter gone bad. A library missing. A full disk. A config issue. An issue forking. Memory limits. All of it.

Configuration issues can be, and often are, temporary. Also, bugs are a thing.

> THAT is where the "good engineering" failed.

I agree with this.

> What you want to do is make 5xx like 4xx, because you feel there should never be any way for a SMTP server to say "No, really, this email address doesn't exist.. don't bother trying again".

Of course there's a semantic difference between "come back later" and "go away forever", I'm just arguing that a client that doesn't blindly trust what every connected system tells it is going to more often successfully achieve its goals than one which does.

Re: Post Mortem of Google Outage on 14 December 2020

#113
post #69

Earlier quoted context omitted.

The linear IOP density model seems clever and logical but is a huge source of headaches because it includes a patently false assumption that IOPs scale in proportion to growth in object size. Performance quotas should be assigned at the object level (block device, file system, bucket) regardless of size.

It’s not because the underlying storage is 1 TiB disk, so you’re getting a fixed allocation that’s shared among many other 50mb/s clients (ie assuming flash and overall transfer rates of ~2gigabytes/, that lets them coex roughly 40 customers on 1 machine without over subscribing)? Isn’t that kind of pricing mode about the only one that would be feasible to implement to make this cost effective? How are you thinking i…

The pedantic answer would be to allow customers to specify both minimum storage space and minimum IOPS. Then charge them for whichever is the larger portion of a drive and provide the space and IOPS that they are then paying for.

This way, if you need 10MB but also 25,000 IOPS, then you'll pay for ~10% of a drive. You'll get your minimum required 25,000 IOPS....and also 200GB or whatever share of the drive is required to get you those IOPS.

At the end of the day, I'm not entirely sure it matters whether the cloud provider breaks it out like this as long as it at least has a little gray UI element under the specified storage space slider that reads out the IOPS to you.

It would be exactly the same as the current situation where customers do this manually. So probably not particularly necessary - why make customers fill in additional fields they might not need to?

I do think cloud providers should make it clear during requisition and read back how many IOPS you're getting, just for clarity.

It does seem like having temporarily high quotas and then throttling back seems to break developer experience. Good deeds and punishment - but in this case there's a solid underlying reason why it has detrimental effects.

Re: Post Mortem of Google Outage on 14 December 2020

#114
post #36

it is interesting that is both cases (recent gmail and this one) it was a "migration": "As part of an ongoing migration of the User ID Service to a new quota system" "An ongoing migration was in effect to update this underlying configuration system" it was not a new feature, not a massive hardware failure, it was migrating part of the working product due to some unclear reason of "best practices". both of those migra…

It's expected the outrage was due to a migration, the majority of test is design to cover new feature/bug fix.

A migration is way more complex, it can touch multiple different components and at Google scale that mean different teams, it can be impossible to test this kind of migration without having a testing platforms as big as prod and if Google do it, maybe all gcp will not be enough.

This "unclear" reason can hide a bigger issue like a security bug fix, or just an important migration to go somewhere.

It look like covid19 hit everything, since it begins, attack increases a lot and security need to step up, really fast. Some manager have trouble to handle the full remote situation, some engineers have trouble too, all that combined can create small hole, so some outrage.

How many migration they do way more complex than this one without issue?

I don't think engineering culture is degrading, but shit happen and in extreme situations we see problems easier.

if this outrage was due by a feature or "massive hardware" , we can be very worrie about it

Re: Post Mortem of Google Outage on 14 December 2020

#115
post #22

Earlier quoted context omitted.

Damn, that's awful. Should lead to a deep questioning of people who claim to be migrating your junk to "best practices" if the old config system worked fine for 15 years and the new one caused a massive dataloss outage on Day 1.

New one gets you promotions. Old one is boring and does not. Guess what people want to work on. Seen it happen too many times.

I read this and identified with it, and then read your username

Re: Post Mortem of Google Outage on 14 December 2020

#116

My favorite quote: "prevent fast implementation of global changes." This is how large organizations become slower compared to small "nimble" companies. Everyone fails, but the noticeability of failure incentivizes large organizations to be more careful.

It doesn’t have to be this way, but that’s partly a matter of culture. By aspiring to present/think/act as a monoplatform, Google risks substantially increasing the blast radius of individual component failure. A global quota system mediating every other service sounds both totally on brand, and also the antithesis of everything I learned about public cloud scaling at AWS. There we made jokes, that weren’t jokes, abo…

I often hear 'aim for elimination of global dependencies', but the reality is that there is no way around global dependencies. AWS STS or IAM is just as global as google's. The difference is that google more often builds with some form of guaranteed read-after-write consistency, while AWS is more often 'fail open'. For example, if you remove a permission from a user in GCP, you are guaranteed consistency within 7 minutes [1], while with AWS IAM, your permissions may be arbitrarily stale. This means that when the GCP IAM database leader fails, all operations will globally fail after 7 minutes, while with AWS IAM, everything continues to work when the leader fails, but as an AWS customer, you can never be sure that some policy change has actually become effective.

In general, AWS more often shifts the harder parts of global distributed systems onto their customers, rather than solving them for their customers, like GCP does. For example, GCP cloud storage (s3 equivalent) and datastore (nosql database) provide strongly consistent operations in multi-region configurations, while dynamodb and s3 have only eventually consistent replication across regions; and google's VPCs, message queues, console VM listings, and loadbalancers are global, while AWS's are regional.

[1] https://cloud.google.com/iam/docs/faq#access_revoke

Re: Post Mortem of Google Outage on 14 December 2020

#117

Earlier quoted context omitted.

The odds that they would happen simultaneously if they’re completely unrelated seem astronomically small, certainly? Both are noted as being related to “ongoing migrations,” though AFAICT not related ones. I would bet there’s a human factor connection- e.g., the day before there was a big meeting where a higher-level management gave multiple ops teams go-ahead on their respective plans, resulting in a multiple potent…

I think it's as simple as a case of the Mondays; you wouldn't roll out a migration like that on a Friday or the weekend, and rolling it out on a Monday gives you the least chances of problem occurring on those dates.

A Monday rollout sounds horrible. Probably the worst day for a rollout.

Re: Post Mortem of Google Outage on 14 December 2020

#118

Earlier quoted context omitted.

It doesn’t have to be this way, but that’s partly a matter of culture. By aspiring to present/think/act as a monoplatform, Google risks substantially increasing the blast radius of individual component failure. A global quota system mediating every other service sounds both totally on brand, and also the antithesis of everything I learned about public cloud scaling at AWS. There we made jokes, that weren’t jokes, abo…

I often hear 'aim for elimination of global dependencies', but the reality is that there is no way around global dependencies. AWS STS or IAM is just as global as google's. The difference is that google more often builds with some form of guaranteed read-after-write consistency, while AWS is more often 'fail open'. For example, if you remove a permission from a user in GCP, you are guaranteed consistency within 7 min…

S3 is strongly consistent. https://aws.amazon.com/s3/consistency/

Which of Google's nosql db provides strong consistency - bigtable? Just confirming

Re: Post Mortem of Google Outage on 14 December 2020

#119
post #22

Earlier quoted context omitted.

Damn, that's awful. Should lead to a deep questioning of people who claim to be migrating your junk to "best practices" if the old config system worked fine for 15 years and the new one caused a massive dataloss outage on Day 1.

New one gets you promotions. Old one is boring and does not. Guess what people want to work on. Seen it happen too many times.

Wouldn't paint all migrations with the same brush

Re: Post Mortem of Google Outage on 14 December 2020

#120

Earlier quoted context omitted.

I often hear 'aim for elimination of global dependencies', but the reality is that there is no way around global dependencies. AWS STS or IAM is just as global as google's. The difference is that google more often builds with some form of guaranteed read-after-write consistency, while AWS is more often 'fail open'. For example, if you remove a permission from a user in GCP, you are guaranteed consistency within 7 min…

S3 is strongly consistent. https://aws.amazon.com/s3/consistency/ Which of Google's nosql db provides strong consistency - bigtable? Just confirming

S3 is newly strongly consistent within a single region since last reinvent or so (google cloud storage has been strongly consistent for much longer). However, the cross-region replication for s3 is based on 'copying' [1] so presumably async and not strongly consistent.

GCP datastore and firestore are strongly consistent nosql databases that are available in multi-region configurations [2].

[1] https://docs.aws.amazon.com/AmazonS3/latest/dev/replication.... [2] https://cloud.google.com/datastore/docs/locations

Post reply on HN