Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Post Mortem of Google Outage on 14 December 2020

status.cloud.google.com

Post reply on HN
Print view View original

101–110 of 209 posts

Re: Post Mortem of Google Outage on 14 December 2020

#101

Post by superkuh »

jrockway wrote: post #18
mehrdadn wrote: post #9

I'm skimming this but so far I still don't understand how any kind of authentication failure can or should lead to an SMTP server returning "this address doesn't exist". Does anyone see an explanation? Edit 1: Oh wow I didn't even realize there were multiple incidents. Thanks! Edit 2 (after reading the Gmail post-mortem): AH! It was a messed up domain name! When this event happened, I said senders need to avoid takin…

Different outage. The Gmail postmortem is linked in another thread, but the gist was that "gmail.com" is a configuration value that can be changed at runtime, and someone changed the configuration. Thus, *@gmail.com stopped being a valid address, and they returned "that mailbox is unavailable". If you don't want to scroll to the other thread, here's the postmortem: https://static.googleusercontent.com/media/www.googl…

In that document they seem to think they have solved the issue as of the 15th. But that is far from true. As of yesterday I was still getting unsubscribed from email lists due to bounces on my @gmail.com account. But that's not the worst.

As of yesterday there are some google email customers like NOAA.gov that cannot receive emails from external mailservers (like my personal domain mailserver I run) because they are now proxying through some "security consultant service" ala mx.us.email.fireeyegov.com which causes the SPF validation to fail because it's no longer the external mailserver's IP that's sending it.

    Received-SPF: fail (google.com: domain of superkuh@superkuh.com does not designate 209.85.219.72 as permitted sender) client-ip=209.85.219.72;
Note that IP, 209.85.219.72, that's not my mailserver's IP, that's an IP that Google owns and use with their new setup to foward email for (some) government accounts.

I've re-signed up for the email lists that gmail's behavior got canceled and subscribed to them with my personal domain/mailserver. It's incredible that a random $5/mo VPS has given me better uptime over the last decade than all of google's infrastructure.

Re: Post Mortem of Google Outage on 14 December 2020

#102

Post by inopinatus »

ridaj wrote: post #97
inopinatus wrote: post #88

Earlier quoted context omitted.

> The infrastructure running the auth service, like any service, is going to have a quota system, whether it's global or not. Don’t assume that is the case. That’s exactly the kind of cultural assumption I’m speaking of. Case in point, I routinely run services without quotas or caps and what have you, scale out for load, and alarm on runaway usage, not service unavailable or quota exceeded. I’d rather take the hit th…

> Don’t assume that is the case. What I mean is, it's going to have limits of some sort, right? The world is finite...

I love riding my bike on winding mountain roads. I’m a lot more careful when there’s no safety barrier. Funny thing is, the consequences of slamming into the barrier by mistakenly taking a tight bend at 90 rather than, say, 50, are just as bad as skidding out off a precipice. And I’ve got the scars to prove it.

Re: Post Mortem of Google Outage on 14 December 2020

#103

Post by fuzzyengineer »

freeone3000 wrote: post #83
vlovich123 wrote: post #77

Earlier quoted context omitted.

It’s not because the underlying storage is 1 TiB disk, so you’re getting a fixed allocation that’s shared among many other 50mb/s clients (ie assuming flash and overall transfer rates of ~2gigabytes/, that lets them coex roughly 40 customers on 1 machine without over subscribing)? Isn’t that kind of pricing mode about the only one that would be feasible to implement to make this cost effective? How are you thinking i…

Fixed cost per IOPS allocated. Essentially, the same thing as before, but without the necessity of you storing large blank objects.

Can't this be addressed by provisioned throughput on EFS?

Re: Post Mortem of Google Outage on 14 December 2020

#104

Post by b112 »

growse wrote: post #90
b112 wrote: post #78

Earlier quoted context omitted.

Rejected twice? When do you retry? 5 seconds later? 5 minutes? An hour? When is the message requeued to send? When do you stop trying, when the remote side has already said "Hey, this user has no account here!". Meanwhile, the end user sent a message to bod@ by accident, instead of bob@, and your mail server keeps retrying to send mail, even when the remote mail server said "Hey! That account doesn't exist!". It's a…

Generally speaking, if you're trying to write a reliable distributed system (and email is a massively distributed system), a good principle to follow is to retry on failure, no matter the failure. Obviously there are edge cases, and obviously you don't just retry every 1ms forever, but to assume that an error that comes back from a system you've called once is both (a) the product of a 100% working non-faulty system…

Email is extremely non-brittle, and everyone following the RFC is one main way it is able to be non-brittle. What is naive here, is that you think things are the way there are, for no reason.

Mail is decades old, built upon millions of hours of work crafting software, RFC standards, and works the way it does, including bounces, to ensure stability.

A 5xx error means 'perm failure'. There are a variety of 5xx class responses, from 'user account deleted' to 'no such domain'. It is, in fact, a working mail system, which responds with 5xx, or 4xx (temp fail) or 2xx (received OK) messages.

A severely broken system is incapable of even responding.

For a mail system to respond with 5xx, when it is internally broken, is 100% a configuration issue. Every MTA on the planet, is designed (eg, postfix, sendmail, etc) to respond with a 4xx TEMP fail message if something is borked. A milter gone bad. A library missing. A full disk. A config issue. An issue forking. Memory limits. All of it.

By default, MTAs are designed to 4xx(tempfail) on those errors. Loads, and loads, and loads of work to ensure that. Code meticulously crafted. This is how good engineering works. This is how the RFC works.

You either get no ACK for your SYN, because it is so borked, or you get a 4xx if it can run, but has a failure condition.. OR someone did something very, very wrong.

I understand your angst, but the real problem isn't the RFC, or the sender not retrying. The problem is:

- the MTA was up and running

- it had an error condition on its back end

- whomever set everything up, didn't taking into account internal failure conditions, and to respond with a 4xx if that was the case

THAT is where the "good engineering" failed.

Mail accounts are deactivated all the time. Domains are deactivated all the time. 5xx tells us this. Someone replying with 5xx, when they did not mean to, has a configuration issue on their end.

That's all there is to it.

I'll put this another way.

What you want to do is make 5xx like 4xx, because you feel there should never be any way for a SMTP server to say "No, really, this email address doesn't exist.. don't bother trying again".

Re: Post Mortem of Google Outage on 14 December 2020

#105

Post by jiggawatts »

ahupp wrote: post #74
jiggawatts wrote: post #63

I've been bitten by quota grace periods before, they're the "buffer bloat" of cloud platform management systems. You build something, it seems fine, and then a month later it keels over. My tiny disaster was caused by Amazon EFS. They provide a performance quota grace period of a month, during which all tests passed with flying colours. Turns out that EFS rations out IOPS per GB of stored data, and this particular ap…

I ran into this same issue with GCP while using the boot disk for some caches. In this case the grace period is a few minutes before they throttle. It was quite a pain to track down.

The "pain to track down" is important because nothing failed.

I didn't get an alert.

There was nothing in the logs.

There wasn't anything in the portal to indicate that something had changed.

Everything was up and responding, just really, really slowly.

The application wasn't even timing out, because the EFS share itself was responding to TCP ACKs instantly, and even the timeouts at the NFS protocol layer weren't being hit. I was just getting one... file... at... a... time.

After about 2-3 hours of troubleshooting I opened a service ticket with AWS, and it took them another few hours to figure out what was going on.

Like other people in this thread suggested, I first copied a 10 GB empty file into the volume to speed it up. Later they made the fixed IOPS SKU available and I switched to that.

Re: Post Mortem of Google Outage on 14 December 2020

#106

Post by b112 »

b112 wrote: post #104
growse wrote: post #90

Earlier quoted context omitted.

Generally speaking, if you're trying to write a reliable distributed system (and email is a massively distributed system), a good principle to follow is to retry on failure, no matter the failure. Obviously there are edge cases, and obviously you don't just retry every 1ms forever, but to assume that an error that comes back from a system you've called once is both (a) the product of a 100% working non-faulty system…

Email is extremely non-brittle, and everyone following the RFC is one main way it is able to be non-brittle. What is naive here, is that you think things are the way there are, for no reason. Mail is decades old, built upon millions of hours of work crafting software, RFC standards, and works the way it does, including bounces, to ensure stability. A 5xx error means 'perm failure'. There are a variety of 5xx class re…

I just realised that you may not get the other side of the scenario.

On a 5xx series response, you bounce. Where does the bounce go? Back to the original sender.

This is the 'closing of the loop'. Original sender sees the message was not sent successfully. They can now retry, resend, make a phone call, whatever.

"Hey Bob, what was your email address again? I got a bounce. Ohhh, damn, I typed bod@ instead of bob@. I'll fix and resend.

That's how it works.

That's how it works for:

- a typo in the email address

- when an account is deleted (and maybe your friend has a new email address)

- the remote mail admin did something horribly, horribly wrong, and it bounced as a result

(Horribly, horribly wrong being -- they told their own mail servers that accounts did not exist, who helpfully passed that info on.)

Re: Post Mortem of Google Outage on 14 December 2020

#107

Post by jrockway »

ridaj wrote: post #97
inopinatus wrote: post #88

Earlier quoted context omitted.

> The infrastructure running the auth service, like any service, is going to have a quota system, whether it's global or not. Don’t assume that is the case. That’s exactly the kind of cultural assumption I’m speaking of. Case in point, I routinely run services without quotas or caps and what have you, scale out for load, and alarm on runaway usage, not service unavailable or quota exceeded. I’d rather take the hit th…

> Don’t assume that is the case. What I mean is, it's going to have limits of some sort, right? The world is finite...

Yes, everything has limits. Where Google says "quota system", for normal people that means "buy another computer"; you have hit your quota when you're out of memory / cpu cycles / disk. At Google, they have some extra computers sitting around, but it's still not infinite. Quota is a way of hitting some sort of limit before every atom in the Universe becomes a computer on which to run your program.

I don't think there is any way to avoid it. It sounds bad when it's software that's telling your service it can't write to disk, rather than the disk not having any more free sectors on which to write, but it's exactly the same thing. Everyone has a quota, and left unchecked, your software will run into it.

(In the case of this postmortem, there was a bug in the software, which makes it all feel self-inflicted. But if it wasn't self-inflicted, the same problem would have manifested in some other way.)

There is a comment in this thread where the author says they take less risks when the safety systems are turned off. That is fine and nice, but is not really a good argument against safety systems. I have definitely had outages where something hit a quota I set, but I've had more confusing outages from something exhausting all physical resources, and an unrelated system failing because it happened to be nearby. I think you should wear a helmet AND ride safely.

Re: Post Mortem of Google Outage on 14 December 2020

#108

Post by ineedasername »

jiggawatts wrote: post #63

I've been bitten by quota grace periods before, they're the "buffer bloat" of cloud platform management systems. You build something, it seems fine, and then a month later it keels over. My tiny disaster was caused by Amazon EFS. They provide a performance quota grace period of a month, during which all tests passed with flying colours. Turns out that EFS rations out IOPS per GB of stored data, and this particular ap…

Even worse are undocumented quotas. Probably not quite the same thing, but I've had such an issue with a SaaS API for extracting data for a cloud system. It has an unpublished limit on queries. Writing a library of code to use it, I was fine. All tests passed. Running a full day's data pull & integration had sporadic failure: It would run w/o issue for half the job, then have seeming random errors on some requests.

I kill the job, start again, and the problem is there from the very beginning. Kill again, review my code for an hour or two (python, grequest) tweak some parameters, start the job again and it seems fine, problem solved? Nope, halfway through the same issue occurs.

More testing showed even that "halfway through" wasn't consistent.

It's the hidden quota, and it wasn't even consistent: Sometimes it would go 10%, sometimes 70%. I don't know if they even had static limits or some type of dynamic system. Then you get throttled, only requests fail without any indicative error message except the "failed" or "unavailable" (I forget which)

I determined the limit was tied to the API key. Given that error messages & the unknown limit made working within their hidden limits difficult, and support was neither sympathetic or forthcoming with details necessary to work within their limits, I simply created multiple API keys & rotated between them on requests. Probably not the best behavior for a tenant of a shared system, but I didn't see much choice when the alternative was to not use an essential advertised feature.

Re: Post Mortem of Google Outage on 14 December 2020

#109

Post by jhgg »

pram wrote: post #91
vlovich123 wrote: post #77

Earlier quoted context omitted.

It’s not because the underlying storage is 1 TiB disk, so you’re getting a fixed allocation that’s shared among many other 50mb/s clients (ie assuming flash and overall transfer rates of ~2gigabytes/, that lets them coex roughly 40 customers on 1 machine without over subscribing)? Isn’t that kind of pricing mode about the only one that would be feasible to implement to make this cost effective? How are you thinking i…

It would make more sense to me to charge for the size of the provisioned share and performance profile, like EBS, instead of this weird performance tiering. It basically makes EFS useless for the bulk of use-cases I can personally imagine.

They do that as well - see "provisioned throughput"

See the section "Specifying Throughput with Provisioned Mode" here: https://docs.aws.amazon.com/efs/latest/ug/performance.html

Re: Post Mortem of Google Outage on 14 December 2020

#110

Post by jhgg »

jiggawatts wrote: post #105
ahupp wrote: post #74

Earlier quoted context omitted.

I ran into this same issue with GCP while using the boot disk for some caches. In this case the grace period is a few minutes before they throttle. It was quite a pain to track down.

The "pain to track down" is important because nothing failed . I didn't get an alert. There was nothing in the logs. There wasn't anything in the portal to indicate that something had changed. Everything was up and responding , just really, really slowly. The application wasn't even timing out, because the EFS share itself was responding to TCP ACKs instantly, and even the timeouts at the NFS protocol layer weren't b…

To be fair, there is a cloud watch metric you could have set an alert on: "PercentIOLimit"
Post reply on HN