Slightly off topic - it would be refreshing if financial and other institutions provided similar public post mortems on incidents that affect large numbers of their clients. Recent ones that come to mind are Interactive Brokers and Robinhood.
Post Mortem of Google Outage on 14 December 2020
191–200 of 209 posts
Re: Post Mortem of Google Outage on 14 December 2020
#192Earlier quoted context omitted.
Same people argue that email is a hot mess You're lumping together "email content" with "SMTP". SMTP isn't a "patchwork of standards", it has a very specific RFC for it. SMTP works very, very, very well. Configuration issues can be, and often are, temporary. Also, bugs are a thing. This isn't just "a bug" or "a config issue", this is an edge case bug or config issue. I've handled literally hundreds of thousands of ma…
> And it's not 'go away forever', it's "this specific mail cannot be delivered, please return it to sender, so they can examine the issue and deal with it". > > It's "bring a human into the equation". > > How is this an issue? For something which is very, very rare. You apparently missed the whole point of newsletters automatically unsubscribing users on a 5xx error from the previous discussion that the topmost comme…
You apparently missed the whole point of newsletters automatically unsubscribing users on a 5xx error from the previous discussion
There was no previous discussion / topmost thread, as I replied directly to the topmost comment, and this thread forks from that. No mention was made of newsletter unsubscribes thread upwards. Further, I made it quite clear I was discussing SMTP response codes, and not the article directly.
Ergo, your stated context, isn't the context of this whole thread/discussion. Further, from what I see, everyone in this thread is discussing MTAs, SMTP, SMTP return codes.
that the topmost comment was referring to. Not that you are to blame for missing that because it wasn't restated here, but that's the context of the whole discussion.
No, it isn't the context of this thread/discussion, as per above.
If I have sent emails successfully to a certain email address before it is not wise to assume a permanent error on a single response that the RFC specifies as permanent. Email/SMTP is in it's core a stateless protocol, but as a mailing list service I can keep state and provide additional context to errors that can improve my service.
In the context of the RFC, mailing list software should be viewed the same as a 'human being'. Of course it's fine to it to re-send, if the software wishes. Just like it is fine for you to send a mail with your mail client, get a 5xx, and re-try by clicking 'send' once again.
(5xx errors can happen during auth/etc stages too)
However, during this entire thread I've have been peppering the words 'MTA', "smtp server', 'bounce' and more.
Note that the mailing list software you're describing, is receiving a bounce. Bounces only happen from the MTA side. EG, mailing list software doesn't "bounce" anything, ever. Only an MTA does. Without that bounce, most mailing list software won't even know there is an issue.
Others upthread were advocating that MTAs don't bounce on an initial 5xx failure, regardless of my assertions that the client/end user should receive an immediate 5xx perm failure message.
Whether a human, or some automated software, as per my upstream statements, the proper thing to do is bounce back to the sender. 5xx, bounce, return. In 99.9999999% of cases, this is what is needed. This is the majority case.
Then, the mailing list software, the end user, can do as they wish. Including trying a resend. I don't see the conflict here, except it's apparent that a lot of people don't have much in the way of MTA experience. That's not even a knock on them, but it is a bit disheartening to see people suggesting massive MTA behaviour/RFC alteration for no reason.
To speak to mailing list behaviour, as a separate issue from above, repeated hits to 5xx targets will get you blacklisted faster than you can imagine. It's the equivalent of knocking on someone's door, them answering and saying "Sorry, Bob moved out", but you come back all day and night, banging on the door "HEY IS BOB THERE?!"
Yeah. That'll work out well. Just because you can do something, doesn't mean you even remotely should.
Re: Post Mortem of Google Outage on 14 December 2020
#193My favorite quote: "prevent fast implementation of global changes." This is how large organizations become slower compared to small "nimble" companies. Everyone fails, but the noticeability of failure incentivizes large organizations to be more careful.
I'm pretty certain that that's a misunderstanding, and rollouts at Google still happen faster than at most other companies incl small & mid size. It's just that Google won't rollout to the whole world in one instant step, instead, it depends: can be hours, days or weeks. Or sometimes a first small step during less than a minute, and then a more large scale deployment — e.g. if there's an urgent security bugfix. It's in the SRE book (well as far as I remember), I think you'll find it if you search for "seconds" or "minutes"
https://static.googleusercontent.com/media/sre.google/en//st...
But yes definitely there are other things that slow down big companies.
Re: Post Mortem of Google Outage on 14 December 2020
#194Earlier quoted context omitted.
Even worse are undocumented quotas. Probably not quite the same thing, but I've had such an issue with a SaaS API for extracting data for a cloud system. It has an unpublished limit on queries. Writing a library of code to use it, I was fine. All tests passed. Running a full day's data pull & integration had sporadic failure: It would run w/o issue for half the job, then have seeming random errors on some requests. I…
There are some "fun" horror stories of people hitting the Azure Resource Manager API quota limits. At least one of them was linked to some sort of shared service principal for the client org, so if anyone hit it, the whole organisation could get locked out. It would first start to throw HTTP 429 codes occasionally, then you would get locked out for exponentially increasing times, up to two weeks or something absurd l…
My team struggled with this for two years. We ended up helping MSFT add a caching layer to the Kubernetes Azure cloud controller and creating a separate Azure subscription for each Kubernetes cluster.
Re: Post Mortem of Google Outage on 14 December 2020
#195I'm skimming this but so far I still don't understand how any kind of authentication failure can or should lead to an SMTP server returning "this address doesn't exist". Does anyone see an explanation? Edit 1: Oh wow I didn't even realize there were multiple incidents. Thanks! Edit 2 (after reading the Gmail post-mortem): AH! It was a messed up domain name! When this event happened, I said senders need to avoid takin…
My understanding from the other threads about the Gmail outage is that Google will assume you're a spammer if you trigger no such mailbox too often (because they think you're generating random addresses and trying them), so they have conditioned other operators to permanently drop such addresses. So they kind of created the whole situation with the opaque spam filter system…
Re: Post Mortem of Google Outage on 14 December 2020
#196I'm skimming this but so far I still don't understand how any kind of authentication failure can or should lead to an SMTP server returning "this address doesn't exist". Does anyone see an explanation? Edit 1: Oh wow I didn't even realize there were multiple incidents. Thanks! Edit 2 (after reading the Gmail post-mortem): AH! It was a messed up domain name! When this event happened, I said senders need to avoid takin…
My understanding from the other threads about the Gmail outage is that Google will assume you're a spammer if you trigger no such mailbox too often (because they think you're generating random addresses and trying them), so they have conditioned other operators to permanently drop such addresses. So they kind of created the whole situation with the opaque spam filter system…
That would make more sense (for others) if that's the case, but I'm kind of skeptical because it would mean they've massively screwed up on that front against their own interests. I would think they would have the common sense to understand that a couple re-delivery attempts to an account that was recently closed is clearly not someone trying to spam.
Re: Post Mortem of Google Outage on 14 December 2020
#197Earlier quoted context omitted.
> And it's not 'go away forever', it's "this specific mail cannot be delivered, please return it to sender, so they can examine the issue and deal with it". > > It's "bring a human into the equation". > > How is this an issue? For something which is very, very rare. You apparently missed the whole point of newsletters automatically unsubscribing users on a 5xx error from the previous discussion that the topmost comme…
You apparently missed the whole point of newsletters automatically unsubscribing users on a 5xx error from the previous discussion There was no previous discussion / topmost thread, as I replied directly to the topmost comment, and this thread forks from that. No mention was made of newsletter unsubscribes thread upwards. Further, I made it quite clear I was discussing SMTP response codes, and not the article directl…
I don't really intend to continue the discussion as most things have been said.
Just wanna point you to the context. The topmost comment was referring to his own comment and discussion about the original issue 3 days ago. You can find this here https://news.ycombinator.com/item?id=25438169
The topmost commenter also replied to you that there was that previous discussion here https://news.ycombinator.com/item?id=25473468
So while you didn't have the context, he and probably many others (like myself) did have that context. Again, this is not to blame you, because as I said it wasn't repeated in this thread and you couldn't have known.
> In the context of the RFC, mailing list software should be viewed the same as a 'human being'. Of course it's fine to it to re-send, if the software wishes. [...]
> Then, the mailing list software, the end user, can do as they wish. Including trying a resend. I don't see the conflict here,[...]
A lot of people were arguing 3 days ago that the end user (e.g. the mailing list software) should never try a resend and removing the email immediately (after the first 550 response) is correct behavior and mandaotry by the RFC.
The topmost comment here restated that this is in fact wrong. Based on what I quoted from you here, you're actually agreeing with the topmost comment you were initially disagreeing with.
Here is the relevant part from the topmost comment:
> When this event happened, I said senders need to avoid taking "invalid address" at face value when they've recently succeeded delivering to the same addresses. But despite the RFC saying senders "should not" repeat requests (rather than "must not"), many people had a lot of resistance to this idea, and instead just blamed Google for messing up implementing the RFC.
Re: Post Mortem of Google Outage on 14 December 2020
#198Earlier quoted context omitted.
You apparently missed the whole point of newsletters automatically unsubscribing users on a 5xx error from the previous discussion There was no previous discussion / topmost thread, as I replied directly to the topmost comment, and this thread forks from that. No mention was made of newsletter unsubscribes thread upwards. Further, I made it quite clear I was discussing SMTP response codes, and not the article directl…
I don't really intend to continue the discussion as most things have been said. Just wanna point you to the context. The topmost comment was referring to his own comment and discussion about the original issue 3 days ago. You can find this here https://news.ycombinator.com/item?id=25438169 The topmost commenter also replied to you that there was that previous discussion here https://news.ycombinator.com/item?id=25473…
Saying I agree with the topmost comment, is not accurate.
What I said, and with the context and nuance indicated in this thread, is different than what the top most comment's author said, and its replies.
Re: Post Mortem of Google Outage on 14 December 2020
#199To be honest, I never understood the point of companies publishing post mortems after outages. What is it supposed to accomplish? We the users don’t understand their infrastructure anyway, how are we expected to understand the post mortem? Besides, as the users why should we care why things went down? The fact is, they did. And that’s the only thing that matters. I don’t feel better knowing WHY I lost my services. I…
Learning. If problems were solved, nobody would write code. That new code or config is being deployed shows these problems (new feature rollouts, migrations, scaled resilience, etc.) are not yet formally solved. As such, things not known will become known — and usually be revealed in prod. Similarly, in technology systems, there’s no such thing as human error, only uncaught error conditions. Postmortems capture learn…
Sure, but it doesn't stop future outages
Re: Post Mortem of Google Outage on 14 December 2020
#200I'm curious about big outages like this in big internet corps. Does anyone know if SREs in Europe fixed the problem or it relied on people in Mountain View? When an outage this big hits do devs get involved? I've worked as an SRE and it sucks to be fixing developer's bugs in the middle of the night.
Yes, SRE teams typically have a sister SRE team in another continent and time zone.
[deleted]