Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Gmail having issues

google.com

Post reply on HN
Print view View original

81–90 of 460 posts

Re: Gmail having issues

#81

Post by erhk »

redwards510 wrote: post #63
firebaze wrote: post #47

As quite a few googlers appear to read and write on HN, I'd really welcome an insider info on what's going on the last few days. Sure there will be some internal turmoil going on right now, but isn't there some non-confidential info to share? Can't imagine this will hurt the image of google neither in the short nor long run, quite the opposite.

In lieu of an actual Googler, how about some educated speculation? It blows my mind that Google can even have problems like this. Aren't their apps highly distributed across tons of CDNs? Don't they have world class Devops people that roll out changes in a piecemeal fashion to check for bugs? How exactly can they have an issue that can affect a huge swath of their customers across countries? Insight appreciated.

Software isn't as simple as splitting across different locations to prevent global failures.

Re: Gmail having issues

#82

Post by [deleted] »

redwards510 wrote: post #63
firebaze wrote: post #47

As quite a few googlers appear to read and write on HN, I'd really welcome an insider info on what's going on the last few days. Sure there will be some internal turmoil going on right now, but isn't there some non-confidential info to share? Can't imagine this will hurt the image of google neither in the short nor long run, quite the opposite.

In lieu of an actual Googler, how about some educated speculation? It blows my mind that Google can even have problems like this. Aren't their apps highly distributed across tons of CDNs? Don't they have world class Devops people that roll out changes in a piecemeal fashion to check for bugs? How exactly can they have an issue that can affect a huge swath of their customers across countries? Insight appreciated.

[deleted]

Re: Gmail having issues

#83

Post by brown9-2 »

redwards510 wrote: post #63
firebaze wrote: post #47

As quite a few googlers appear to read and write on HN, I'd really welcome an insider info on what's going on the last few days. Sure there will be some internal turmoil going on right now, but isn't there some non-confidential info to share? Can't imagine this will hurt the image of google neither in the short nor long run, quite the opposite.

In lieu of an actual Googler, how about some educated speculation? It blows my mind that Google can even have problems like this. Aren't their apps highly distributed across tons of CDNs? Don't they have world class Devops people that roll out changes in a piecemeal fashion to check for bugs? How exactly can they have an issue that can affect a huge swath of their customers across countries? Insight appreciated.

You don’t really have to speculate, they disclosed yesterday that yesterday’s issue had to do with the automated quota system deciding the auth system had zero quota:

https://status.cloud.google.com/incident/zall/20013#20013003

Re: Gmail having issues

#84

Post by nimchimpsky »

redwards510 wrote: post #63
firebaze wrote: post #47

As quite a few googlers appear to read and write on HN, I'd really welcome an insider info on what's going on the last few days. Sure there will be some internal turmoil going on right now, but isn't there some non-confidential info to share? Can't imagine this will hurt the image of google neither in the short nor long run, quite the opposite.

In lieu of an actual Googler, how about some educated speculation? It blows my mind that Google can even have problems like this. Aren't their apps highly distributed across tons of CDNs? Don't they have world class Devops people that roll out changes in a piecemeal fashion to check for bugs? How exactly can they have an issue that can affect a huge swath of their customers across countries? Insight appreciated.

That isn't educated speculation, its just throwing stones.

Re: Gmail having issues

#85

Post by belorn »

coldpie wrote: post #43
Seirdy wrote: post #41

Earlier quoted context omitted.

This is another good reason to have email @yourowndomain.tld When this happens, you can spin up a temporary server and have a mechanism in place to redirect email so you don't go down when your provider does.

I've had way more downtime trying to run my own domain's mailserver for a year than I have with gmail for more than a decade.

Having run my own mail server for over a decade, I have yet to have a single time where the server responds by Permanent error of accounts not existing and with email bouncing.

Losing incoming email is pretty much the worst case scenario when it come to configuration errors. It about as bad as not having backups, in that both cases results in unrecoverable loss of data.

Re: Gmail having issues

#86

Post by ab8 »

rat9988 wrote: post #62
firebaze wrote: post #47

As quite a few googlers appear to read and write on HN, I'd really welcome an insider info on what's going on the last few days. Sure there will be some internal turmoil going on right now, but isn't there some non-confidential info to share? Can't imagine this will hurt the image of google neither in the short nor long run, quite the opposite.

I would advice anyone to not share any information that his company hasn't agreed explicitely to share.

Your username is rat9988. Been burned in the past?

Re: Gmail having issues

#87

Post by [deleted] »

redwards510 wrote: post #63
firebaze wrote: post #47

As quite a few googlers appear to read and write on HN, I'd really welcome an insider info on what's going on the last few days. Sure there will be some internal turmoil going on right now, but isn't there some non-confidential info to share? Can't imagine this will hurt the image of google neither in the short nor long run, quite the opposite.

In lieu of an actual Googler, how about some educated speculation? It blows my mind that Google can even have problems like this. Aren't their apps highly distributed across tons of CDNs? Don't they have world class Devops people that roll out changes in a piecemeal fashion to check for bugs? How exactly can they have an issue that can affect a huge swath of their customers across countries? Insight appreciated.

[deleted]

Re: Gmail having issues

#88

Post by User23 »

firebaze wrote: post #47

As quite a few googlers appear to read and write on HN, I'd really welcome an insider info on what's going on the last few days. Sure there will be some internal turmoil going on right now, but isn't there some non-confidential info to share? Can't imagine this will hurt the image of google neither in the short nor long run, quite the opposite.

The question is what exactly is the new “feature” that got pushed skipping canary is.

Re: Gmail having issues

#89

Post by megous »

wenbin wrote: post #9

"Type: Permanent; SubType: General; Code: smtp; 550-5.1.1 The email account that you tried to reach does not exist. Please try 550-5.1.1 double-checking the recipient's email address for typos or 550-5.1.1 unnecessary spaces. Learn more at 550 5.1.1 https://support.google.com/mail/?p=NoSuchUser y128si147264pfg.177 - gsmtp"

A lot of people will lose transactional email messages, because of this.

I'd absolutely hate to be hit by this at this time. Thankfully I've made an time investment to run my own mail server years ago. A handful of times it broke down, it either went offline or started returning 4xx codes due to misconfigured or broken milter after an update. Neither meant lost messages from normal senders that use queuing MTAs.

Re: Gmail having issues

#90

Post by ravenstine »

redwards510 wrote: post #63
firebaze wrote: post #47

As quite a few googlers appear to read and write on HN, I'd really welcome an insider info on what's going on the last few days. Sure there will be some internal turmoil going on right now, but isn't there some non-confidential info to share? Can't imagine this will hurt the image of google neither in the short nor long run, quite the opposite.

In lieu of an actual Googler, how about some educated speculation? It blows my mind that Google can even have problems like this. Aren't their apps highly distributed across tons of CDNs? Don't they have world class Devops people that roll out changes in a piecemeal fashion to check for bugs? How exactly can they have an issue that can affect a huge swath of their customers across countries? Insight appreciated.

Maybe they have world class DevOps, but they also have way more things that can go wrong than the vast majority of businesses. It's kind of remarkable that the entire world can be pinging Google services and they have ~99.9% uptime.
Post reply on HN