Will there be a public post mortem?
Gmail having issues
301–310 of 460 posts
Re: Gmail having issues
#302CEO of an email marketing platform here (EmailOctopus). If anyone's curious, here's a chart showing our bounce rate to Gmail addresses over the course of the week: https://pbs.twimg.com/media/EpUE20UXYAEa_Uv?format=jpg&name=... That's a peak of 90% of Gmail inboxes bouncing – and this has been going on for almost 24 hours.
I know this is your livelihood, but as someone who basically never wants marketing emails, all I can think is "nice" hopefully I get auto-unsub'ed from a ton of lists.
Re: Gmail having issues
#303I'm glad to have found this thread, but its kept me up cleaning out thousands of "bounced" blocks from the last 2 day's suppression lists. They are both @gmail.com and gSuite/Workplace domains, and incidentally a ton of random blocks from Comcast as well from users that have months of open history. This blew away about 10% of our newsletter and marketing subscribers. I can't imagine the time you're having if you send…
Have you considered the possibility these "attacks" occured and were detected months before but were only announced recently? Could it be that some people have a motive to spin some stories during these politically sensitive times?
Re: Gmail having issues
#304Earlier quoted context omitted.
This is the Nightmare Scenario for mailing lists. Many of them auto-unsubscribe after a bounce.
I said this in another comment but this seems like a naive way to react to an "address does not exist error" that they've already delivered to before. The only legit scenario in which that happens is when the user deletes the address, which is a rare event (pretty much always The underlying issue (wherever this occurs) seems to be lack of nuance regarding error codes when people try to implement robust systems. Diffe…
Aka throw the RFC out of the window and implement a broken system because Google did that?
Re: Gmail having issues
#305Earlier quoted context omitted.
This is pretty much the worst response possible. Hard bounces mean that email delivery services are going to start automatically removing, or at least stopping delivery to, entire slews of email addresses. A lot of clean up is going to be needed as a result of this. To add some more details, when using a 3rd party email delivery service, those services will either black-list or just outright remove email addresses wh…
I really cannot believe they did not immediately hack in a new rule to their SMTP server: never return a 5xx (permanent failure), instead return a 421 (temporary failure try again later). That simple fix buys them 24-72 hours to solve this properly. Yeah, it burdens servers sending mail to them because now they have to hold on to all mail (including mail that really is permanently undeliverable) for another day or so…
This outage seems to have lasted for about 2.5 hours. Probably this was fixed by rolling back whatever caused it. (I don't think the rollout was finished before they resolved it; my mail server sends a lot of emails to Gmail addresses, and even at peak I was only seeing maybe about 1/3 mails be rejected.)
There is no way that putting in a hardcored hack like that would have been faster. Making the change is, of course, fast.
But then you need to review it (and this is a super risky change, so the review can't be rubber stamped). Build a production build and run all your qualification tests. (Hope you found all the tests that depend on permanent errors being signalled properly). And then roll it out globally, which again is a risky operation, but with the additional problem that rolling restarts simply can't be done faster than a certain speed since you can only restart so many processes at once while still continuing to serve traffic.
The kind of thing you describe simply can't be done by changing the SMTP server, in 2.5 hours. The best you could get is if there was some kind of abuse or security related articulation point in the system, with fast pushes as required by the problem domain but still with the sufficient power to either prevent the requests from reaching the SMTP server at all, or intercept and change the response.
As a trivial example, something like blocking the SMTP port with a firewall rule could have been viable. Though it has the cost of degrading performance for everyone rather than just the affected requests.
Re: Gmail having issues
#306Earlier quoted context omitted.
How do the public gmail addresses compare to the enterprise (used to be G Suite, now Google Workspace) ones?
I would be very interested to know this as well. I am trying to switch my company over to Google Workspace right now and support has been telling me my signup issues will be "resolved in 48 hours or less." What a joke. And this after we're leaving AWS Workmail because of bounced emails. No luck with signing up so far.
SLA of less than 99,5%... Or if there is multiple issues even sub 99%... That really is a joke...
Re: Gmail having issues
#307I'm glad to have found this thread, but its kept me up cleaning out thousands of "bounced" blocks from the last 2 day's suppression lists. They are both @gmail.com and gSuite/Workplace domains, and incidentally a ton of random blocks from Comcast as well from users that have months of open history. This blew away about 10% of our newsletter and marketing subscribers. I can't imagine the time you're having if you send…
Have you considered the possibility these "attacks" occured and were detected months before but were only announced recently? Could it be that some people have a motive to spin some stories during these politically sensitive times?
Ya that’s quite possible, and probable given how well they are documented to the senate. Dizzying times we live in. The flow of information is dammed and diverted by so many different actors with different motives.
Still, I have good enough reasons to think that some sort of Pandora’s box of backdoors was opened this fall and its fallout is yet to be felt.
Re: Gmail having issues
#308Earlier quoted context omitted.
This is the Nightmare Scenario for mailing lists. Many of them auto-unsubscribe after a bounce.
I said this in another comment but this seems like a naive way to react to an "address does not exist error" that they've already delivered to before. The only legit scenario in which that happens is when the user deletes the address, which is a rare event (pretty much always The underlying issue (wherever this occurs) seems to be lack of nuance regarding error codes when people try to implement robust systems. Diffe…
> I said this in another comment but this seems like a naive way to react to an "address does not exist error" that they've already delivered to before.
Like HTTP, SMTP is also designed to be stateless so, in the first place, the remote server shouldn't return a permanent error in temporary failure scenarios.
The default error should be 450: "Requested action not taken – The user’s mailbox is unavailable”, not "the user has deleted everything and left".
These standards worked well before big players came and told "My responses tell what I chose them to say, and these meaning doesn't always overlap with the established standards". The only exception is spam and we now have standards for helping to reduce it.
Re: Gmail having issues
#309Earlier quoted context omitted.
but not all at once
Also, never ever a 550 5.1.1 error.
On the same server... Was missing an email, went to look into other box that should have forwarded that. Got very suspicious looking email...
Good thing gmail wasn't completely stupid and didn't try to forward that one, getting in infinite loop...
Re: Gmail having issues
#310CEO of an email marketing platform here (EmailOctopus). If anyone's curious, here's a chart showing our bounce rate to Gmail addresses over the course of the week: https://pbs.twimg.com/media/EpUE20UXYAEa_Uv?format=jpg&name=... That's a peak of 90% of Gmail inboxes bouncing – and this has been going on for almost 24 hours.
As an ESP, how much of a headache will this be for you in weeks/months to come? I'm guessing this throws a huge wrench in deliverability techniques--how're you handling it?
It's a real headache but should be fully reversible. @shmoogy hit the nail on the head: we'll run through our events in that timeframe, inspect the raw bounce reason to check it relates to the Gmail outage, then undo the actions that the bounce caused.
The reason why this is so nasty is not because Gmail went down, but because they returned a 5XX permanent failure and not a 4XX temporary failure for these bounces. Literally every email provider will respond to a permanent bounce by suppressing all further emails to that email address (it's permanent, after all!), so the fallout from this will be huge.