Post Mortem of Google Outage on 14 December 2020 • Hacker News Board

Re: Post Mortem of Google Outage on 14 December 2020

#181

Post by erhk » Sat, Dec 19, 2020, 1:37 PM UTC

inopinatus wrote: post #44

physicsgraph wrote: post #11

My favorite quote: "prevent fast implementation of global changes." This is how large organizations become slower compared to small "nimble" companies. Everyone fails, but the noticeability of failure incentivizes large organizations to be more careful.

It doesn’t have to be this way, but that’s partly a matter of culture. By aspiring to present/think/act as a monoplatform, Google risks substantially increasing the blast radius of individual component failure. A global quota system mediating every other service sounds both totally on brand, and also the antithesis of everything I learned about public cloud scaling at AWS. There we made jokes, that weren’t jokes, abo…

If Google is flaky you use yahoo.

If Facebook/Twitter/instagram is flaky you wait until it isn't and then post that update.

Re: Post Mortem of Google Outage on 14 December 2020

#182

Post by erhk » Sat, Dec 19, 2020, 1:39 PM UTC

inopinatus wrote: post #128

felixhuttmann wrote: post #116

Earlier quoted context omitted.

I often hear 'aim for elimination of global dependencies', but the reality is that there is no way around global dependencies. AWS STS or IAM is just as global as google's. The difference is that google more often builds with some form of guaranteed read-after-write consistency, while AWS is more often 'fail open'. For example, if you remove a permission from a user in GCP, you are guaranteed consistency within 7 min…

> In general, AWS more often shifts the harder parts of global distributed systems onto their customers, rather than solving them for their customers, like GCP does. Choice of language in representing this is rather telling, because AWS can (and does) pitch this as a strength, viz. that regionalisation helps customers (especially, significantly, bigco enterprise customers) reason about the possible failure modes, and…

Slow rollouts are a security hole.

Re: Post Mortem of Google Outage on 14 December 2020

#183

Post by ascar » Sat, Dec 19, 2020, 1:43 PM UTC

b112 wrote: post #147

growse wrote: post #112

Earlier quoted context omitted.

> Email is extremely non-brittle, and everyone following the RFC is one main way it is able to be non-brittle. What is naive here, is that you think things are the way there are, for no reason. > Mail is decades old, built upon millions of hours of work crafting software, RFC standards, and works the way it does, including bounces, to ensure stability. Same people argue that email is a hot mess exactly because it's d…

Same people argue that email is a hot mess You're lumping together "email content" with "SMTP". SMTP isn't a "patchwork of standards", it has a very specific RFC for it. SMTP works very, very, very well. Configuration issues can be, and often are, temporary. Also, bugs are a thing. This isn't just "a bug" or "a config issue", this is an edge case bug or config issue. I've handled literally hundreds of thousands of ma…

> And it's not 'go away forever', it's "this specific mail cannot be delivered, please return it to sender, so they can examine the issue and deal with it". > > It's "bring a human into the equation". > > How is this an issue? For something which is very, very rare.

You apparently missed the whole point of newsletters automatically unsubscribing users on a 5xx error from the previous discussion that the topmost comment was referring to. Not that you are to blame for missing that because it wasn't restated here, but that's the context of the whole discussion. In this context I also heavily disagree with your statements and agree that among other examples automatically unsubscribing to a single "permanent" error is a brittle system.

If I have sent emails successfully to a certain email address before it is not wise to assume a permanent error on a single response that the RFC specifies as permanent. Email/SMTP is in it's core a stateless protocol, but as a mailing list service I can keep state and provide additional context to errors that can improve my service.

Finally, if you read the RFC 5321 4.2.1. Reply Code Severities and Theory it specifically uses "SHOULD NOT" instead of "MUST NOT" implying that there might be valid scenarios to act differently.

RFC 2119:

> SHOULD NOT This phrase, or the phrase "NOT RECOMMENDED" mean that there may exist valid reasons in particular circumstances when the particular behavior is acceptable or even useful, but the full implications should be understood and the case carefully weighed before implementing any behavior described with this label.

Re: Post Mortem of Google Outage on 14 December 2020

#184

Post by ascar » Sat, Dec 19, 2020, 1:48 PM UTC

b112 wrote: post #70

mehrdadn wrote: post #9

I'm skimming this but so far I still don't understand how any kind of authentication failure can or should lead to an SMTP server returning "this address doesn't exist". Does anyone see an explanation? Edit 1: Oh wow I didn't even realize there were multiple incidents. Thanks! Edit 2 (after reading the Gmail post-mortem): AH! It was a messed up domain name! When this event happened, I said senders need to avoid takin…

This is how mail works. Follow the RFC or you're wrong. https://tools.ietf.org/html/rfc5321 I haven't read the article, but SMTP response codes are very specific. If there were SMTP response codes in the 5xx range, that's a perm failure, end of story. Temp fail messages, 4xx, can be used for 'try later'. That's how SMTP works. What you are suggesting would break email.

> If there were SMTP response codes in the 5xx range, that's a perm failure, end of story.

This is, plain and simply, wrong. The RFC does not state that.

RFC 5321 4.2.1 Reply Code Severities and Theory

> 5yz Permanent Negative Completion reply

> The command was not accepted and the requested action did not occur. The SMTP client SHOULD NOT repeat the exact request (in the same sequence). Even some "permanent" error conditions can be corrected, so the human user may want to direct the SMTP client to reinitiate the command sequence by direct action at some point in the future (e.g., after the spelling has been changed, or the user has altered the account status)

SHOULD NOT is also clearly defined by RFC 2119:

> This phrase, or the phrase "NOT RECOMMENDED" mean that there may exist valid reasons in particular circumstances when the particular behavior is acceptable or even useful, but the full implications should be understood and the case carefully weighed before implementing any behavior described with this label.

Re: Post Mortem of Google Outage on 14 December 2020

#185

Post by b112 » Sat, Dec 19, 2020, 1:54 PM UTC

jiggawatts wrote: post #172

b112 wrote: post #150

Earlier quoted context omitted.

I'll be the fly in the ointment here, the additional sand in your shorts, and say "This is why bare metal is better". Bare metal doesn't have to be VM/containerless. Roll your own. But at least in this case, you're dealing with your own issues, with things not hidden and abstracted away, and at literally 1/100th to 1/10000th the cost of AWS. And yes, that's with the hardware investment and wage costs rolled in. I fee…

Hardware has the exact same monitoring issues, worse even. The dinosaurs of the on-prem hardware world like Dell, HPE and IBM make the most atrocious systems management software that I have ever seen. Bargain basement quality at best. The cloud is eating their lunch for a reason. Everyone wants a single pane of glass, not a hundred unique and special vendor-specific consoles to manage one app. Everyone wants unified…

With bare metal, you just have to roll your own monitoring.

Primarily, only buy what you can work with easily. Mainly, esure raid monitoring can be scripted, and you really need not depend upon the manufacturer's horrid software (which I agree is just that). A bit of IPMI for failed power supplies, or what not. If a box dies, it dies, that's what failover is for.

I don't your logging complain, you need it no matter what you do. Just use an rsyslog server, logcheck or other app, monitor for important events, done. I've never used anything off the shelf, and yet all I hear are complaints from those that do.

I get how people might want to eschew the above for simple stuff, but once you start needing heavy monitoring, it's all the same ball of wax.

Re: Post Mortem of Google Outage on 14 December 2020

#186

Post by b112 » Sat, Dec 19, 2020, 2:07 PM UTC

ascar wrote: post #184

b112 wrote: post #70

Earlier quoted context omitted.

This is how mail works. Follow the RFC or you're wrong. https://tools.ietf.org/html/rfc5321 I haven't read the article, but SMTP response codes are very specific. If there were SMTP response codes in the 5xx range, that's a perm failure, end of story. Temp fail messages, 4xx, can be used for 'try later'. That's how SMTP works. What you are suggesting would break email.

> If there were SMTP response codes in the 5xx range, that's a perm failure, end of story. This is, plain and simply, wrong. The RFC does not state that. RFC 5321 4.2.1 Reply Code Severities and Theory > 5yz Permanent Negative Completion reply > The command was not accepted and the requested action did not occur. The SMTP client SHOULD NOT repeat the exact request (in the same sequence). Even some "permanent" error c…

The RFC does state that. Note the use of 'future' and 'corrected'.

Understand context. The perm I state, is referencing that SMTP session, yet at the end of the session, your job is to bounce back to end user.

Else, how can the human fix things "after the spelling has been changed"? How can the human get involved without the bounce?

Re: Post Mortem of Google Outage on 14 December 2020

#187

Post by shaicoleman » Sat, Dec 19, 2020, 2:16 PM UTC

jugg1es wrote: post #126

ineedasername wrote: post #108

Earlier quoted context omitted.

Even worse are undocumented quotas. Probably not quite the same thing, but I've had such an issue with a SaaS API for extracting data for a cloud system. It has an unpublished limit on queries. Writing a library of code to use it, I was fine. All tests passed. Running a full day's data pull & integration had sporadic failure: It would run w/o issue for half the job, then have seeming random errors on some requests. I…

Ooh boy. Undocumented limitations caused major problems with us with AWS SQS. It is now in the documentation (maybe because I complained to our AWS rep), but SQS has pool of 20k messages that it will pull from when serving requests. If you are using a FIFO queue and you have 20k messages with the same message group ID in that buffer, then you are unable to process any other message regardless of how large the queue i…

Pretty scary that it just breaks without any error messages, and without an easy way to clean up the damage.

Queue overflows are not uncommon.

"For FIFO queues, there can be a maximum of 20,000 inflight messages (received from a queue by a consumer, but not yet deleted from the queue). If you reach this quota, Amazon SQS returns no error messages. If your queue has a large backlog of 20,000 or more messages with the same message group ID, FIFO queues might be unable to return the messages that have a different message group ID but were sent to the queue at a later time until you successfully consume the messages from the backlog."

Link to the documentation: https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQS...

Re: Post Mortem of Google Outage on 14 December 2020

#188

Post by Terretta » Sat, Dec 19, 2020, 2:34 PM UTC

felixhuttmann wrote: post #116

inopinatus wrote: post #44

Earlier quoted context omitted.

It doesn’t have to be this way, but that’s partly a matter of culture. By aspiring to present/think/act as a monoplatform, Google risks substantially increasing the blast radius of individual component failure. A global quota system mediating every other service sounds both totally on brand, and also the antithesis of everything I learned about public cloud scaling at AWS. There we made jokes, that weren’t jokes, abo…

I often hear 'aim for elimination of global dependencies', but the reality is that there is no way around global dependencies. AWS STS or IAM is just as global as google's. The difference is that google more often builds with some form of guaranteed read-after-write consistency, while AWS is more often 'fail open'. For example, if you remove a permission from a user in GCP, you are guaranteed consistency within 7 min…

AWS’s regionality and stronger region separation boundaries are a huge selling point for regulated (data regulation) industries and enterprises.

A bank, for instance, may be required to prove it cannot replicate customer data across regions, and that no third party provider will replicate its data using their own BCM or DR systems.

Regardless of CSP, startups should think about rules on movement of data among data jurisdictions (such as GDPR) and architect accordingly.

Re: Post Mortem of Google Outage on 14 December 2020

#189

Post by Terretta » Sat, Dec 19, 2020, 2:45 PM UTC

TedShiller wrote: post #160

To be honest, I never understood the point of companies publishing post mortems after outages. What is it supposed to accomplish? We the users don’t understand their infrastructure anyway, how are we expected to understand the post mortem? Besides, as the users why should we care why things went down? The fact is, they did. And that’s the only thing that matters. I don’t feel better knowing WHY I lost my services. I…

Learning.

If problems were solved, nobody would write code. That new code or config is being deployed shows these problems (new feature rollouts, migrations, scaled resilience, etc.) are not yet formally solved. As such, things not known will become known — and usually be revealed in prod.

Similarly, in technology systems, there’s no such thing as human error, only uncaught error conditions.

Postmortems capture learning, so the conditions can be caught next time.

Publishing them shows the engineering organization understands and applies this learning loop.

Re: Post Mortem of Google Outage on 14 December 2020

#190

Post by bogomipz » Sat, Dec 19, 2020, 3:28 PM UTC

jiggawatts wrote: post #63

I've been bitten by quota grace periods before, they're the "buffer bloat" of cloud platform management systems. You build something, it seems fine, and then a month later it keels over. My tiny disaster was caused by Amazon EFS. They provide a performance quota grace period of a month, during which all tests passed with flying colours. Turns out that EFS rations out IOPS per GB of stored data, and this particular ap…

>""The baseline rate is 50 MiB/s per TiB of storage (equivalently, 50 KiB/s per GiB of storage)."

Does the mean that baseline is only realized when at least a Tib of storage is actually being used then? In other words there was a distinction between how much you storage you were actually using vs how much storage you provisioned? Are there other services that use this same model?