Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Gmail having issues

google.com

Post reply on HN
Print view View original

231–240 of 460 posts

Re: Gmail having issues

#231

Post by toomuchtodo »

dathinab wrote: post #214
toomuchtodo wrote: post #201

Earlier quoted context omitted.

Being down is okay. Returning an error message that results in the data being thrown away instead of being requeued is not. Block incoming smtp connections until your app layer is fixed.

> Block incoming smtp connections until you app layer is fixed. Or returning one of the 4xx status codes which indicate less-permanent failure state like: - 451 Requested action aborted: local error in processing Which is kinda like a HTTP internal server error as it can mean anything.

For my comment’s purposes, I assume if this was possible with a flag or config setting (and the code path existed), it would’ve already been done. Doesn’t seem like they can, so they should’ve pulled the handbrake and gone “full stop” without throwing everyone’s mail away (hence blocking incoming connections and let the mail sit in all of the external MTA queues).

Another option would’ve been to accept everything with a very lightweight smtp ingest service, journal it all, and play it back to the full frontend after their code fix was pushed out.

Not an SRE so ¯\_(ツ)_/¯ just some thoughts from my time in a similar role and similar pain points (but thankfully not at this scale)

Re: Gmail having issues

#232

Post by rob-olmos »

shmoogy wrote: post #189
whitepoplar wrote: post #185

Earlier quoted context omitted.

As an ESP, how much of a headache will this be for you in weeks/months to come? I'm guessing this throws a huge wrench in deliverability techniques--how're you handling it?

I would imagine since it's a known timeframe, domain, and error response, they can cleanly remove the suppression lists. I logged into our sendgrid and mailgun accounts and manually purged all the failed gmail records.

Might also be affecting GSuite/Workspace emails.

Re: Gmail having issues

#233

Post by nine_k »

Karunamon wrote: post #56
coldpie wrote: post #43

Earlier quoted context omitted.

I've had way more downtime trying to run my own domain's mailserver for a year than I have with gmail for more than a decade.

Use a paid email host, just anything but Google. Life's too short to put up with managing your own email server.

It can as well be Google, just the paid Apps version. Zero time to get used to a different UI. I suspect there must be a solution to easily migrate all your tags and filtering rules. (Tags are the killer feature to me. Outlook sort of has them but they are less flexible.)

Re: Gmail having issues

#234

Post by ttul »

Email sender here (on behalf of about 2.5 million domains). We noticed the issue and mitigated it by transforming the 5xx error into a 4xx error so that messages to Google are re-queues instead of being permfailed. But even with this intervention, the ticket volume was insane...

Re: Gmail having issues

#236

Post by brewdad »

Tenoke wrote: post #177
NautilusWave wrote: post #172

Earlier quoted context omitted.

I bet it adjusts for time zone automatically, as I see: > We will provide an update by 12/15/20, 5:30 PM

I hate that even more as I can never know if they adjusted it correctly or not, even more so if I've traveled recently or used a VPN.

This is Google. They know where you've been. They know where you're going before you do.

I agree that a uniform time zone would be clearer.

Re: Gmail having issues

#237

Post by dataflow »

octoberfranklin wrote: post #209
windexh8er wrote: post #195

Just got this from the ProtonMail team: > Dear ProtonMail user, Starting at around 4:30PM New York (10:30PM Zurich), Gmail suffered a global outage. A catastrophic failure at Gmail is causing emails sent to Gmail to permanently fail and bounce back. The error message from Gmail is the following: 550-5.1.1 The email account that you tried to reach does not exist. This is a global issue, and it impacts all email provid…

This is the Nightmare Scenario for mailing lists. Many of them auto-unsubscribe after a bounce.

I said this in another comment but this seems like a naive way to react to an "address does not exist error" that they've already delivered to before. The only legit scenario in which that happens is when the user deletes the address, which is a rare event (pretty much always The underlying issue (wherever this occurs) seems to be lack of nuance regarding error codes when people try to implement robust systems. Different codes imply different things and shouldn't all just fall back into generic buckets.

Re: Gmail having issues

#239

Post by mcqueenjordan »

dataflow wrote: post #237
octoberfranklin wrote: post #209

Earlier quoted context omitted.

This is the Nightmare Scenario for mailing lists. Many of them auto-unsubscribe after a bounce.

I said this in another comment but this seems like a naive way to react to an "address does not exist error" that they've already delivered to before. The only legit scenario in which that happens is when the user deletes the address, which is a rare event (pretty much always The underlying issue (wherever this occurs) seems to be lack of nuance regarding error codes when people try to implement robust systems. Diffe…

Good callout. The underlying issue of the lack of nuance is probably /state/. Being more nuanced about these errors probably requires managing state, which tends to increase the complexity and scaling challenges.

Re: Gmail having issues

#240

Post by blibble »

dannyw wrote: post #218
blibble wrote: post #142

workspace (gsuite) customers: remember to claim under the SLA, cos they don't award it automatically under 99.9% is a 3 day credit, and they're currently at 99.4%

I sent in a support ticket. Is that the right way to claim SLA?

not sure, but that's what I've done too
Post reply on HN