Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Gmail having issues

google.com

Post reply on HN
Print view View original

191–200 of 460 posts

Re: Gmail having issues

#191

Post by Andrex »

missblit wrote: post #161
joatmon-snoo wrote: post #91

Earlier quoted context omitted.

Googler but nowhere near Gmail, so just educated speculation: * We have a lot of automation/tools to prevent incidents when mitigation is straightforward (e.g. roll back a bad flag, quarantine unusual traffic patterns), which means that when something does go wrong it's often a new failure mode that needs custom, specialized mitigation. (e.g. what if you're in a situation where rolling back could make the problem wor…

Rollback proof bugs are rare, but boy howdy are they exciting. I think I've only seen one so far (unless you count bad data / bad state that persists after a bad change is rolled back... which can also be pretty exciting)

Is "exciting" a synonym for "harrowing" where you're from? :P

Re: Gmail having issues

#192

Post by Aperocky »

joatmon-snoo wrote: post #91
redwards510 wrote: post #63

Earlier quoted context omitted.

In lieu of an actual Googler, how about some educated speculation? It blows my mind that Google can even have problems like this. Aren't their apps highly distributed across tons of CDNs? Don't they have world class Devops people that roll out changes in a piecemeal fashion to check for bugs? How exactly can they have an issue that can affect a huge swath of their customers across countries? Insight appreciated.

Googler but nowhere near Gmail, so just educated speculation: * We have a lot of automation/tools to prevent incidents when mitigation is straightforward (e.g. roll back a bad flag, quarantine unusual traffic patterns), which means that when something does go wrong it's often a new failure mode that needs custom, specialized mitigation. (e.g. what if you're in a situation where rolling back could make the problem wor…

> what if you're in a situation where rolling back could make the problem worse?

Here comes the poison pills!

Re: Gmail having issues

#193

Post by throwaway201103 »

edoceo wrote: post #181
jonathanbull wrote: post #153

CEO of an email marketing platform here (EmailOctopus). If anyone's curious, here's a chart showing our bounce rate to Gmail addresses over the course of the week: https://pbs.twimg.com/media/EpUE20UXYAEa_Uv?format=jpg&name=... That's a peak of 90% of Gmail inboxes bouncing – and this has been going on for almost 24 hours.

What's this do to your mail-queue size - let's see that chart

Permanent failures, as these are being flagged, don't stay in the queue.

Re: Gmail having issues

#194

Post by csnover »

pmlnr wrote: post #106
Uehreka wrote: post #95

Earlier quoted context omitted.

No, designers did not get lazy. Devices got weirder. It used to be that you could check the width and height of the viewport and say something like “320px wide? Must be a touch interface, deploy the big buttons”. Then tablets got big and it was like “1024px? Could be a laptop, but it’s probably an iPad, which has a touch interface, deploy the big buttons”. Then laptops got touch screens, then the Surface Studio came…

Simple solution would be to ask the user what they want. I genuinely don't understand why this is not common instead of trying to guess it.

You don’t have to ask the user, there is a media rule for querying whether the device is currently using coarse or fine pointer input[0] (though, of course, it relies on the OS not lying, which is not a given).

[0] https://developer.mozilla.org/en-US/docs/Web/CSS/@media/poin...

Re: Gmail having issues

#195

Post by windexh8er »

Just got this from the ProtonMail team:

> Dear ProtonMail user,

Starting at around 4:30PM New York (10:30PM Zurich), Gmail suffered a global outage.

A catastrophic failure at Gmail is causing emails sent to Gmail to permanently fail and bounce back. The error message from Gmail is the following:

550-5.1.1 The email account that you tried to reach does not exist.

This is a global issue, and it impacts all email providers trying to send email to Gmail, not just ProtonMail.

Because Gmail is sending a permanent failure, our mail servers will not automatically retry sending these messages (this is standard practice at all email services for handling permanent failures).

We are closely monitoring the situation. At this time, little can be done until Google fixes the problem. We recommend attempting to resend the messages to Gmail users when Google has fixed the problem. You can find the latest status from Google's status page:

https://www.google.com/appsstatus#hl=en&v=issue&sid=1&iid=a8...

Best Regards, The ProtonMail Team

Re: Gmail having issues

#196

Post by Netcob »

AnssiH wrote: post #30
j-wags wrote: post #20

Over the past 24 hours, I've had GitHub request that I re-verify my gmail three times (roughly 22 hours ago, 2 hours ago, and now), each time resetting my primary email's status to "Undeliverable" and "Unverified" The triggering event may be an email bounce. I get a lot of github notifications sent to my email, and the failure of just one/a few may trigger the reverification.

Yep, there was a very similar event yesterday, approx. 22 hours ago: https://www.google.com/appsstatus#hl=en&v=issue&sid=1&iid=10...

When that happened I panicked a little, realizing how much Google Sheets data I had that wasn't really backed up anywhere since Sheets files in Google Drive are basically just links. I started a Takeout, but it looks like I wasn't the only one - it took well over a day to complete.

Re: Gmail having issues

#197

Post by jrockway »

megous wrote: post #115
erhk wrote: post #81

Earlier quoted context omitted.

Software isn't as simple as splitting across different locations to prevent global failures.

I thought SMTP was specifically designed for this (with support for multiple MX entries, queuing at the sender MTA side, etc.) and there's an easy hard boundary at the user mailbox level you can use to partition your system. It should not be a problem that gmail is "down". Unless this would be happening for more than a few days, noone would lose e-mail. It's a problem that it's not returning a temporary error code, b…

It is pretty clear that accepting a TCP connection and reading the bytes of the email from the sender is not the problem. Google is bouncing messages with an error like "that user doesn't exist". This would lead one to believe that some instances are having trouble looking up users, and that doesn't scale super easily. If the product guarantees that it will reject invalid email addresses (which is nice of them, not required by any spec), there has to be a globally consistent record of what email addresses are valid, and the accepting server has to look it up and react in the time that the sender is still connected to the mail server. You can't queue those and send the bounce later (there is no reliable "from" field in email; the only way to correctly bounce is while the sender is still connected). This basically means that you have on the order of milliseconds to accept or reject the email, so merely starting up a another replica of your SMTP daemon isn't going to mitigate this issue. The chokepoint is querying the list of users to see if you should bounce or accept the email. They made it hard on themselves by providing messages like "that user doesn't exist", but... it is nice when you email someone and you get the message "they got fired, sorry" instead of silence. So they made their system more complicated than it needed to be, for a better user experience, and now they are fighting a breakage.

Re: Gmail having issues

#198

Post by m3nu »

whitepoplar wrote: post #185
jonathanbull wrote: post #153

CEO of an email marketing platform here (EmailOctopus). If anyone's curious, here's a chart showing our bounce rate to Gmail addresses over the course of the week: https://pbs.twimg.com/media/EpUE20UXYAEa_Uv?format=jpg&name=... That's a peak of 90% of Gmail inboxes bouncing – and this has been going on for almost 24 hours.

As an ESP, how much of a headache will this be for you in weeks/months to come? I'm guessing this throws a huge wrench in deliverability techniques--how're you handling it?

Mailgun added a few new suppressions due to bounced Gmail addresses. Hope ESPs just flush those out.

Re: Gmail having issues

#199

Post by systemvoltage »

windexh8er wrote: post #195

Just got this from the ProtonMail team: > Dear ProtonMail user, Starting at around 4:30PM New York (10:30PM Zurich), Gmail suffered a global outage. A catastrophic failure at Gmail is causing emails sent to Gmail to permanently fail and bounce back. The error message from Gmail is the following: 550-5.1.1 The email account that you tried to reach does not exist. This is a global issue, and it impacts all email provid…

This feels like a cheap shot at Google. Shit happens, and they're not immune to it even if the servers are located in Zurich. Running a datacenter is no easy task.

Re: Gmail having issues

#200

Post by EarthIsHome »

snazz wrote: post #159
alphadevx wrote: post #135

Earlier quoted context omitted.

Same for me, mainly for privacy concerns. And I back it up daily to my local NAS. It's so easy to configure and run your own mail server, that I'm surprised we are the minority in the tech community.

> It's so easy to configure and run your own mail server Is it? Is dealing with IP reputation, getting your emails accepted by major providers, and being on the hook for fixing everything yourself very easy? I haven't tried, so I don't have personal experience, but I've heard enough horror stories to think that it's not a good use of my time.

I had an IP reputation issue and managed to resolve it after some time.

TLDR: Before you spin up a mail server, check if your IP address is on any of the blacklists [0]-[1] as well as Proof Point's list [2]. If it is, then try and get a different IP address.

I spun up a hosted server on Digital Ocean and received an IP address. I checked several black lists from a few email testing/troubleshooting sites [0] and [1] and all was groovy; my IP address wasn't on any list.

I got a bunch of 521 bounces when I tried emailing a neighbor who had an att.net address.

So, I checked the troubleshooting websites, and my IP address was listed as clean.

My logs said I should forward the error to abuse_rbl@abuse-att.net, so I did.

Those emails were never delivered, because abuse-att.net had its own blacklist. I was getting 553 errors. In the logs, the message from their server told me to check https://ipcheck.proofpoint.com.

Proof point runs their own blacklist that some enterprises use (e.g. att and apple [3]). I checked their list, and lo and behold, my IP address from Digital Ocean was blocked [2]. Digital Ocean wasn't able to remove the IP address from their blocklist and suggested I spin up a new droplet with a different IP address.

I didn't want to do that, so I sent Proof Point an email that went unanswered; the email asked them to remove my IP address. I forgot about the issue for five or six months (this is a personal server), and ran into the issue again a few months ago. So I sent Proof Point an email again, this time with different wording emphasizing that "my clients" were having delivery issues. Within a day, they removed my IP address from their block list.

So, my main suggestion is to check if your IP address is on any of the blacklists as well as Proof Point's list before you start on your server. If it is, then try and get a different IP address.

Does anyone have more "enterprise" lists, like Proof Point, to check?

[0]: https://www.mail-tester.com/

[1]: https://mxtoolbox.com/blacklists.aspx

[2]: https://ipcheck.proofpoint.com

[3]: https://www.reddit.com/r/email/comments/6toxzr/ip_blocked_by...

Post reply on HN