I've been bitten by quota grace periods before, they're the "buffer bloat" of cloud platform management systems. You build something, it seems fine, and then a month later it keels over. My tiny disaster was caused by Amazon EFS. They provide a performance quota grace period of a month, during which all tests passed with flying colours. Turns out that EFS rations out IOPS per GB of stored data, and this particular ap…
Post Mortem of Google Outage on 14 December 2020
111–120 of 209 posts
Re: Post Mortem of Google Outage on 14 December 2020
#112Earlier quoted context omitted.
Generally speaking, if you're trying to write a reliable distributed system (and email is a massively distributed system), a good principle to follow is to retry on failure, no matter the failure. Obviously there are edge cases, and obviously you don't just retry every 1ms forever, but to assume that an error that comes back from a system you've called once is both (a) the product of a 100% working non-faulty system…
Email is extremely non-brittle, and everyone following the RFC is one main way it is able to be non-brittle. What is naive here, is that you think things are the way there are, for no reason. Mail is decades old, built upon millions of hours of work crafting software, RFC standards, and works the way it does, including bounces, to ensure stability. A 5xx error means 'perm failure'. There are a variety of 5xx class re…
> Mail is decades old, built upon millions of hours of work crafting software, RFC standards, and works the way it does, including bounces, to ensure stability.
Same people argue that email is a hot mess exactly because it's decades old and comprised of a patchwork of standards. It's not exactly the poster child of how to do this sort of thing well.
> For a mail system to respond with 5xx, when it is internally broken, is 100% a configuration issue. Every MTA on the planet, is designed (eg, postfix, sendmail, etc) to respond with a 4xx TEMP fail message if something is borked. A milter gone bad. A library missing. A full disk. A config issue. An issue forking. Memory limits. All of it.
Configuration issues can be, and often are, temporary. Also, bugs are a thing.
> THAT is where the "good engineering" failed.
I agree with this.
> What you want to do is make 5xx like 4xx, because you feel there should never be any way for a SMTP server to say "No, really, this email address doesn't exist.. don't bother trying again".
Of course there's a semantic difference between "come back later" and "go away forever", I'm just arguing that a client that doesn't blindly trust what every connected system tells it is going to more often successfully achieve its goals than one which does.
Re: Post Mortem of Google Outage on 14 December 2020
#113Earlier quoted context omitted.
The linear IOP density model seems clever and logical but is a huge source of headaches because it includes a patently false assumption that IOPs scale in proportion to growth in object size. Performance quotas should be assigned at the object level (block device, file system, bucket) regardless of size.
It’s not because the underlying storage is 1 TiB disk, so you’re getting a fixed allocation that’s shared among many other 50mb/s clients (ie assuming flash and overall transfer rates of ~2gigabytes/, that lets them coex roughly 40 customers on 1 machine without over subscribing)? Isn’t that kind of pricing mode about the only one that would be feasible to implement to make this cost effective? How are you thinking i…
This way, if you need 10MB but also 25,000 IOPS, then you'll pay for ~10% of a drive. You'll get your minimum required 25,000 IOPS....and also 200GB or whatever share of the drive is required to get you those IOPS.
At the end of the day, I'm not entirely sure it matters whether the cloud provider breaks it out like this as long as it at least has a little gray UI element under the specified storage space slider that reads out the IOPS to you.
It would be exactly the same as the current situation where customers do this manually. So probably not particularly necessary - why make customers fill in additional fields they might not need to?
I do think cloud providers should make it clear during requisition and read back how many IOPS you're getting, just for clarity.
It does seem like having temporarily high quotas and then throttling back seems to break developer experience. Good deeds and punishment - but in this case there's a solid underlying reason why it has detrimental effects.
Re: Post Mortem of Google Outage on 14 December 2020
#114it is interesting that is both cases (recent gmail and this one) it was a "migration": "As part of an ongoing migration of the User ID Service to a new quota system" "An ongoing migration was in effect to update this underlying configuration system" it was not a new feature, not a massive hardware failure, it was migrating part of the working product due to some unclear reason of "best practices". both of those migra…
A migration is way more complex, it can touch multiple different components and at Google scale that mean different teams, it can be impossible to test this kind of migration without having a testing platforms as big as prod and if Google do it, maybe all gcp will not be enough.
This "unclear" reason can hide a bigger issue like a security bug fix, or just an important migration to go somewhere.
It look like covid19 hit everything, since it begins, attack increases a lot and security need to step up, really fast. Some manager have trouble to handle the full remote situation, some engineers have trouble too, all that combined can create small hole, so some outrage.
How many migration they do way more complex than this one without issue?
I don't think engineering culture is degrading, but shit happen and in extreme situations we see problems easier.
if this outrage was due by a feature or "massive hardware" , we can be very worrie about it
Re: Post Mortem of Google Outage on 14 December 2020
#115Earlier quoted context omitted.
Damn, that's awful. Should lead to a deep questioning of people who claim to be migrating your junk to "best practices" if the old config system worked fine for 15 years and the new one caused a massive dataloss outage on Day 1.
New one gets you promotions. Old one is boring and does not. Guess what people want to work on. Seen it happen too many times.
Re: Post Mortem of Google Outage on 14 December 2020
#116My favorite quote: "prevent fast implementation of global changes." This is how large organizations become slower compared to small "nimble" companies. Everyone fails, but the noticeability of failure incentivizes large organizations to be more careful.
It doesn’t have to be this way, but that’s partly a matter of culture. By aspiring to present/think/act as a monoplatform, Google risks substantially increasing the blast radius of individual component failure. A global quota system mediating every other service sounds both totally on brand, and also the antithesis of everything I learned about public cloud scaling at AWS. There we made jokes, that weren’t jokes, abo…
In general, AWS more often shifts the harder parts of global distributed systems onto their customers, rather than solving them for their customers, like GCP does. For example, GCP cloud storage (s3 equivalent) and datastore (nosql database) provide strongly consistent operations in multi-region configurations, while dynamodb and s3 have only eventually consistent replication across regions; and google's VPCs, message queues, console VM listings, and loadbalancers are global, while AWS's are regional.
Re: Post Mortem of Google Outage on 14 December 2020
#117Earlier quoted context omitted.
The odds that they would happen simultaneously if they’re completely unrelated seem astronomically small, certainly? Both are noted as being related to “ongoing migrations,” though AFAICT not related ones. I would bet there’s a human factor connection- e.g., the day before there was a big meeting where a higher-level management gave multiple ops teams go-ahead on their respective plans, resulting in a multiple potent…
I think it's as simple as a case of the Mondays; you wouldn't roll out a migration like that on a Friday or the weekend, and rolling it out on a Monday gives you the least chances of problem occurring on those dates.
Re: Post Mortem of Google Outage on 14 December 2020
#118Earlier quoted context omitted.
It doesn’t have to be this way, but that’s partly a matter of culture. By aspiring to present/think/act as a monoplatform, Google risks substantially increasing the blast radius of individual component failure. A global quota system mediating every other service sounds both totally on brand, and also the antithesis of everything I learned about public cloud scaling at AWS. There we made jokes, that weren’t jokes, abo…
I often hear 'aim for elimination of global dependencies', but the reality is that there is no way around global dependencies. AWS STS or IAM is just as global as google's. The difference is that google more often builds with some form of guaranteed read-after-write consistency, while AWS is more often 'fail open'. For example, if you remove a permission from a user in GCP, you are guaranteed consistency within 7 min…
Which of Google's nosql db provides strong consistency - bigtable? Just confirming
Re: Post Mortem of Google Outage on 14 December 2020
#119Earlier quoted context omitted.
Damn, that's awful. Should lead to a deep questioning of people who claim to be migrating your junk to "best practices" if the old config system worked fine for 15 years and the new one caused a massive dataloss outage on Day 1.
New one gets you promotions. Old one is boring and does not. Guess what people want to work on. Seen it happen too many times.
Re: Post Mortem of Google Outage on 14 December 2020
#120Earlier quoted context omitted.
I often hear 'aim for elimination of global dependencies', but the reality is that there is no way around global dependencies. AWS STS or IAM is just as global as google's. The difference is that google more often builds with some form of guaranteed read-after-write consistency, while AWS is more often 'fail open'. For example, if you remove a permission from a user in GCP, you are guaranteed consistency within 7 min…
S3 is strongly consistent. https://aws.amazon.com/s3/consistency/ Which of Google's nosql db provides strong consistency - bigtable? Just confirming
GCP datastore and firestore are strongly consistent nosql databases that are available in multi-region configurations [2].
[1] https://docs.aws.amazon.com/AmazonS3/latest/dev/replication.... [2] https://cloud.google.com/datastore/docs/locations