Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Gmail having issues

google.com

Post reply on HN
Print view View original

431–440 of 460 posts

Re: Gmail having issues

#431

Post by 8fingerlouie »

dmm wrote: post #418
8fingerlouie wrote: post #414

Earlier quoted context omitted.

> People will suggest running your own mail server, and if you have the time and energy then definitely do that. As a learning experience, sure, but most people are not prepared for what running a 24/7 mail service requires of them. First of all, a static, non-residential IP is likely needed. The big players will flat out refuse receiption if your IP is registered as residential, so that rules out hosting it from you…

> First of all, a static, non-residential IP is likely needed. If you want to directly send mail that's true. But if you send mail through a smarthost, like your isp's smtp server, you can easily receive mail on a dynamic, residential ip. > implement the infrastructure to actually host a server 24/7, email is really tolerant of downtime. You can be down for hours without losing mail. The sending servers will retry fo…

> email is really tolerant of downtime. You can be down for hours without losing mail. The sending servers will retry for a while.

I’m aware senders will retry for days if your server doesn’t reply, but it still requires monitoring and is not just a “fire and forget” solution.

Also, if your server starts bouncing emails, chances are you’ll be missing mails. Again, needs monitoring.

Re: Gmail having issues

#432

Post by jsnell »

BubuIIC wrote: post #427
jsnell wrote: post #305

Earlier quoted context omitted.

This outage seems to have lasted for about 2.5 hours. Probably this was fixed by rolling back whatever caused it. (I don't think the rollout was finished before they resolved it; my mail server sends a lot of emails to Gmail addresses, and even at peak I was only seeing maybe about 1/3 mails be rejected.) There is no way that putting in a hardcored hack like that would have been faster. Making the change is, of cours…

This has been going on for 2 days, not 2 hours.

The linked status page shows a 2.5 hour duration.

My mail server logs show about 20 failures in all of the last week until yesterday 20:43 CET, then 350 failures between 20:43-00:21, then nothing after that. So fair enough, from the client side rather than the status page it looks like 3.5 hours rather than 2.5.

But still, given that resolution time, the suggested solution of changing the SMTP server is absolutely ludicrous.

Re: Gmail having issues

#433

Post by ocdtrekkie »

imhoguy wrote: post #349
ocdtrekkie wrote: post #242

Earlier quoted context omitted.

I see two problems here: The likelihood your service is restored before you spin up your own mail server, and the fact that, not expecting this failure, their DNS may have a fairly lengthy TTL.

What about permanent problem, like suspended account?

In that case, owning your own domain is golden. I just don't see "spin up your own mail server" as a short term solution.

Re: Gmail having issues

#434

Post by schmappel »

8fingerlouie wrote: post #414
gpmcadam wrote: post #353

People will suggest running your own mail server, and if you have the time and energy then definitely do that. But the next best thing you can do is simply just use your own domain. That way, you can at least decide to migrate your email elsewhere. Don't use the free domains you get from things like gmail or other providers, because then you have to _change_ your email address, and not just your MX records.

> People will suggest running your own mail server, and if you have the time and energy then definitely do that. As a learning experience, sure, but most people are not prepared for what running a 24/7 mail service requires of them. First of all, a static, non-residential IP is likely needed. The big players will flat out refuse receiption if your IP is registered as residential, so that rules out hosting it from you…

> I purchased a lifetime subscription (limited promo offer) with mxroute.com.

At the risk of stating the obvious, note that 'lifetime' refers to the lifetime of the company, not the customer. Which underscores the risk of buying lifetime subscriptions.

And as much as I like the idea of avoiding recurring costs (I have a 'lifetime' Plex pass), it seems to me that these can't be sustainable for the company on the long term.

Re: Gmail having issues

#435

Post by upofadown »

dmm wrote: post #418
8fingerlouie wrote: post #414

Earlier quoted context omitted.

> People will suggest running your own mail server, and if you have the time and energy then definitely do that. As a learning experience, sure, but most people are not prepared for what running a 24/7 mail service requires of them. First of all, a static, non-residential IP is likely needed. The big players will flat out refuse receiption if your IP is registered as residential, so that rules out hosting it from you…

> First of all, a static, non-residential IP is likely needed. If you want to directly send mail that's true. But if you send mail through a smarthost, like your isp's smtp server, you can easily receive mail on a dynamic, residential ip. > implement the infrastructure to actually host a server 24/7, email is really tolerant of downtime. You can be down for hours without losing mail. The sending servers will retry fo…

I think you can be down for 4 days by the standard.

Re: Gmail having issues

#436

Post by webbie917 »

jonathanbull wrote: post #153

CEO of an email marketing platform here (EmailOctopus). If anyone's curious, here's a chart showing our bounce rate to Gmail addresses over the course of the week: https://pbs.twimg.com/media/EpUE20UXYAEa_Uv?format=jpg&name=... That's a peak of 90% of Gmail inboxes bouncing – and this has been going on for almost 24 hours.

Thanks for sharing Jonathan, unprecedented situation. And that's just gmail.com addresses we can see data on, while there are all those business domains that use Google Apps for their email that probably experienced a similar issue...

Re: Gmail having issues

#437

Post by webbie917 »

vxNsr wrote: post #302
jonathanbull wrote: post #153

CEO of an email marketing platform here (EmailOctopus). If anyone's curious, here's a chart showing our bounce rate to Gmail addresses over the course of the week: https://pbs.twimg.com/media/EpUE20UXYAEa_Uv?format=jpg&name=... That's a peak of 90% of Gmail inboxes bouncing – and this has been going on for almost 24 hours.

I know this is your livelihood, but as someone who basically never wants marketing emails, all I can think is "nice" hopefully I get auto-unsub'ed from a ton of lists.

This would affect all email types including emails like receipts, shipment confirmations, password resets, account verification.

Plenty of critical communications get caught in this storm...

Re: Gmail having issues

#438

Post by Sebguer »

croon wrote: post #331
Sebguer wrote: post #207

Earlier quoted context omitted.

Yeah, this is a particularly pernicious failure given how email works. Many mailing providers will just mark these as blacklisted, now, and lots of unsophisticated users likely won't notice.

I consider myself sophisticated enough, but my Bitwarden has 700 accounts, of which ~30% old ones are registered with a gmail address, and the rest are handled behind g suite. Granted that last bit might be partly my fault, even though I paid for it. But even for a "sophisticated" user, I have no easy way of knowing if any of these accounts have silently failed to function now, other than by the passage of time and e…

Oh, absolutely, even for sophisticated users mitigating may be difficult or impossible depending on exactly what bounced and how. But you at least are aware that this happened, and that you have a problem. Think how many people are out there with no clue what this error meant, or that it signaled an ecosystem problem, or that just had hundreds or thousands of emails silently bounce and unsubscribe.

Re: Gmail having issues

#439

Post by webbie917 »

whitepoplar wrote: post #185
jonathanbull wrote: post #153

CEO of an email marketing platform here (EmailOctopus). If anyone's curious, here's a chart showing our bounce rate to Gmail addresses over the course of the week: https://pbs.twimg.com/media/EpUE20UXYAEa_Uv?format=jpg&name=... That's a peak of 90% of Gmail inboxes bouncing – and this has been going on for almost 24 hours.

As an ESP, how much of a headache will this be for you in weeks/months to come? I'm guessing this throws a huge wrench in deliverability techniques--how're you handling it?

+1 what Jonathan said. Typically, when email service providers are down the response code indicates a temporary issue with a soft bounce code, so you can still try to send to that address in the future.

The action for rectifying isn't too difficult, but the implications are still pretty big...

Re: Gmail having issues

#440

Post by webbie917 »

shmoogy wrote: post #189
whitepoplar wrote: post #185

Earlier quoted context omitted.

As an ESP, how much of a headache will this be for you in weeks/months to come? I'm guessing this throws a huge wrench in deliverability techniques--how're you handling it?

I would imagine since it's a known timeframe, domain, and error response, they can cleanly remove the suppression lists. I logged into our sendgrid and mailgun accounts and manually purged all the failed gmail records.

The hard bounce status might be stored outside of your lists. I am not sure customers can easily change a hard bounce status themselves. Do you mean you just deleted those records with intent to re-add to reset the status? On our BigMailer platform this wouldn't work as hard bounce status would get preserved.
Post reply on HN