Still getting errors sending transactional messages to users on gmail
Apr 17 10:23:30 postfix/smtp: connect to gmail-smtp-in.l.google.com[2607:f8b0:4002:c02::1a]:25: Network is unreachable
Google Apps Services Were Down
71–80 of 80 posts
Re: Google Apps Services Were Down
#72And We're up.
Re: Google Apps Services Were Down
#73I don't get why apps and all sorts of other google sites go down a lot, but search never has in my experience. Why is that? Is search just simpler? Does it fail in non-obvious ways? Enormous redundancy/fail-over on search?
Re: Google Apps Services Were Down
#74This is the message: "Sorry, there seems to be a problem. The service you're looking for is temporarily unavailable. We're working hard to restore your access as soon as possible. Please try again in a few hours. Thanks for your patience." 0900 CDT edit: works for me now.
That error message seems to just be Google's default 500 error message page; even when it is an API request that fails, even if the API is specified to return errors as some kind of JSON or XML, you get that HTML page in response. (I know this as Google's OpenID+OAuth pipeline has been failing occassionally for the last month, something I've been trying to get them to fix on the Google Federated Login API mailing list.)
Re: Google Apps Services Were Down
#75I don't get why apps and all sorts of other google sites go down a lot, but search never has in my experience. Why is that? Is search just simpler? Does it fail in non-obvious ways? Enormous redundancy/fail-over on search?
Contributing the vast majority of revenue for the company has something to do with it.
Re: Google Apps Services Were Down
#76This is the message: "Sorry, there seems to be a problem. The service you're looking for is temporarily unavailable. We're working hard to restore your access as soon as possible. Please try again in a few hours. Thanks for your patience." 0900 CDT edit: works for me now.
so many developers hacking on google glass apps?
Re: Google Apps Services Were Down
#77The Register reports that GMail went down exactly a year ago: http://www.theregister.co.uk/2012/04/18/google_gmail_outage/
Huffington Post reports that Gmail also went offline on April 29th of 2010.
http://www.huffingtonpost.com/2010/04/29/gmail-down-check-gm...
Google reports that Gmail also went offline on April 24th of 2009.
http://googleblog.blogspot.com/2009/02/current-gmail-outage....
(Sadly, the partial Gmail outage of 2011 seemed to be February 27th, or this would be an epic pattern of "don't trust Gmail during the end of April" ;P.)
http://www.eweek.com/c/a/Messaging-and-Collaboration/Google-...
Re: Google Apps Services Were Down
#78Why do people report this kind of stuff on hacker news. I am sure everyone knows how to google for a site's status page.
Re: Google Apps Services Were Down
#79Earlier quoted context omitted.
Me too, I was really scared. Now I know is Google's problem I am breading again.
> Me too, I was really scared. Now I know is Google's problem I am breading again. I hate it when Google get in the way of my sandwich making ;-)
Given the naming scheme of Android releases, it seems like Google would let you eat cake instead.
Re: Google Apps Services Were Down
#80I don't get why apps and all sorts of other google sites go down a lot, but search never has in my experience. Why is that? Is search just simpler? Does it fail in non-obvious ways? Enormous redundancy/fail-over on search?
Search is (mostly) stateless. They can copy the search database to hundreds of DCs around the world, and each can operate independently - so if one of them has a problem, it's not likely to affect the others (unless it's bad data, but that's why you shouldn't deploy index updates to all DCs simultaneously). There's some personalization, sure, but you can just fall back to non-personalized search if that breaks.
Things like gmail, on the other hand, are inherently stateful. When you log into gmail, you have to eventually connect to one system that maintains your mailbox. Sure, there might be replication - but the replicas are all talking to each other. It's surprisingly easy to have a cascade failure in a system like this, where one of the replicas going down triggers (directly or indirectly) all the others failing as well. Or you can have some bad data that gets replicated out, and then proceeds to confuse everything that's looking at it - unlike search, you have to replicate that data immediately, and don't get to enjoy the benefits of a staged deployment.
This also explains why not all users were affected - I'd guess that their system is divided into some number of shards, and users are assigned to a particular shard. That 0.07% of users affected probably represents a single unhealthy shard.