Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Gmail having issues

google.com

Post reply on HN
Print view View original

161–170 of 460 posts

Re: Gmail having issues

#161

Post by missblit »

joatmon-snoo wrote: post #91
redwards510 wrote: post #63

Earlier quoted context omitted.

In lieu of an actual Googler, how about some educated speculation? It blows my mind that Google can even have problems like this. Aren't their apps highly distributed across tons of CDNs? Don't they have world class Devops people that roll out changes in a piecemeal fashion to check for bugs? How exactly can they have an issue that can affect a huge swath of their customers across countries? Insight appreciated.

Googler but nowhere near Gmail, so just educated speculation: * We have a lot of automation/tools to prevent incidents when mitigation is straightforward (e.g. roll back a bad flag, quarantine unusual traffic patterns), which means that when something does go wrong it's often a new failure mode that needs custom, specialized mitigation. (e.g. what if you're in a situation where rolling back could make the problem wor…

Rollback proof bugs are rare, but boy howdy are they exciting. I think I've only seen one so far (unless you count bad data / bad state that persists after a bad change is rolled back... which can also be pretty exciting)

Re: Gmail having issues

#162

Post by gkop »

jonathanbull wrote: post #153

CEO of an email marketing platform here (EmailOctopus). If anyone's curious, here's a chart showing our bounce rate to Gmail addresses over the course of the week: https://pbs.twimg.com/media/EpUE20UXYAEa_Uv?format=jpg&name=... That's a peak of 90% of Gmail inboxes bouncing – and this has been going on for almost 24 hours.

How do the public gmail addresses compare to the enterprise (used to be G Suite, now Google Workspace) ones?

Re: Gmail having issues

#164

Post by yudlejoza »

redwards510 wrote: post #63
firebaze wrote: post #47

As quite a few googlers appear to read and write on HN, I'd really welcome an insider info on what's going on the last few days. Sure there will be some internal turmoil going on right now, but isn't there some non-confidential info to share? Can't imagine this will hurt the image of google neither in the short nor long run, quite the opposite.

In lieu of an actual Googler, how about some educated speculation? It blows my mind that Google can even have problems like this. Aren't their apps highly distributed across tons of CDNs? Don't they have world class Devops people that roll out changes in a piecemeal fashion to check for bugs? How exactly can they have an issue that can affect a huge swath of their customers across countries? Insight appreciated.

I didn't know being a programming contest script kiddie and grinding leetcode all day automagically makes you a world class Devops, just cz you got into Google.

Re: Gmail having issues

#165

Post by lostmsu »

Imagine surprise reading this after a whole day of productive freelance-related email exchanges that went seemingly without issues, and wondering if all important stuff was received.

By the end of the thread I wondered why it was not affected, until I remembered my small business email is actually on Exchange 365 :-D

Re: Gmail having issues

#166

Post by frongpik »

Karunamon wrote: post #56
coldpie wrote: post #43

Earlier quoted context omitted.

I've had way more downtime trying to run my own domain's mailserver for a year than I have with gmail for more than a decade.

Use a paid email host, just anything but Google. Life's too short to put up with managing your own email server.

I switched to a custom domain only when gmail torpedoed one of my secondary gmail accounts.

Re: Gmail having issues

#167

Post by [deleted] »

mlyle wrote: post #130
LinuxBender wrote: post #48

Earlier quoted context omitted.

They should be returning 421 for backend outages so that sending servers queue and retry the emails. 550 can be interpreted by some as deleted [1] or even banned accounts in some cases. Maybe someone here could convince them to change the logic that occurs during an outage. [1] - https://en.wikipedia.org/wiki/List_of_SMTP_server_return_cod...

Yah. Maybe there's an unexpected way that things can fail resulting in 550's. But maybe at Google's scale you should have some kind of kill switch to stop answering SMTP or to not send permanent errors at all, so that you could flip a switch and prevent the worst consequences of this rather than let it go on for a couple of hours.

[deleted]

Re: Gmail having issues

#168

Post by Havoc »

Side note - why don't google status updates contain a timezone. You'd think that they'd have some awareness of you know...global business...many users in many timezones.

> We will provide an update by 12/15/20, 10:30 PM

Re: Gmail having issues

#169

Post by throwaway201103 »

icedchai wrote: post #138
throwaway201103 wrote: post #99

Earlier quoted context omitted.

His and her no longer correlates to biological sex, so what does it matter anymore. Pronouns can mean whatever you want them to mean.

Okay, so use "their." It is gender neutral, so should work for everyone.

It's also wrong. because it's not singular. Makes for difficult reading.

Re: Gmail having issues

#170

Post by kleinsch »

firebaze wrote: post #47

As quite a few googlers appear to read and write on HN, I'd really welcome an insider info on what's going on the last few days. Sure there will be some internal turmoil going on right now, but isn't there some non-confidential info to share? Can't imagine this will hurt the image of google neither in the short nor long run, quite the opposite.

I don’t work at Google, I’m at a different big tech that’s in the news frequently. Sharing inside info on an ongoing incident is a great way to get fired. Big tech companies are way different than startups where everyone can do a bit of anything. There are people whose job it is to handle that communication. You make their job a lot harder if you disclose information. The company is so big that as an engineer you may not know all the factors involved in what would hurt the company long term - undisclosed relevant litigation, compliance commitments, partner obligations, etc.

How much do you hate it as an engineer when sales people make tech promises to customers without asking you? For comms people, engineers leaking info publicly feels the same way.

Post reply on HN