2000 points in an hour on HN. 120K upvotes on down detector. Trending worldwide with 500K tweets.
Google outage – resolved
651–660 of 870 posts
Re: Google outage – resolved
#652So, anybody still feel like arguing that 'the cloud' is a viable back-up? Or is that a sore point right now? Just for a moment imagine: what if it never comes back again? Of course it will, - at least, it better - but what if it doesn't? And if it does, are you going to take countermeasures in case it happens again or is it just going to be 'back to normal' again?
People would probably argue a “multi cloud” solution. Have your infrastructure be “cloud agnostic” and this sort of problem would be avoided. There was actually a project called “spinnaker” that was supposed to solve this problem. Whether the cost of paying 2 or more cloud providers is worth it for most companies is up in the air.
"Multi-cloud" only works if you stick with the basics. Like disk storage, compute, and a well-supported database. Once you tie in into a cloud's specific offerings.....
Re: Google outage – resolved
#653When I tried to login it said: 'this account does not exist'. So my first thought was some algorithm made a mistake and my account got deleted for no reason. I already imagined the only solution now was to write a medium post and hope it gets some traction on hackernews and google support steps in. Thinking to myself I was an idiot for knowing all this and still thinking it wouldn't happen to me. And even though it t…
Those hosting with their own domains on Google mail had an identical experience, if it makes you feel differently.
Re: Google outage – resolved
#6544:41AM PT, Google services have been restored to my accounts (free & gsuite). And I have never seen them load so fast before - gmail progress bar barely seen for a fraction of a second whereas I am more used to seeing it for multiple seconds (2-3 sec) until it loads. I observe the same anecdotal speedup for other sites... drive, youtube, calendar. I wonder if they are throwing all the hardware they have at their serv…
Everything is snappier for a while if you turn it off and then on again
Re: Google outage – resolved
#655FYI if you have cronjobs using Google apis, disable them or delay them.
a bot is flagging ips as abusive. it should clear up later.
Re: Google outage – resolved
#656Not a fan or even their user but kudos to Google for admitting the failure. Amazon (at least AWS) customers would have probably been shown a green dashboard and told to review their code.
Re: Google outage – resolved
#6574:41AM PT, Google services have been restored to my accounts (free & gsuite). And I have never seen them load so fast before - gmail progress bar barely seen for a fraction of a second whereas I am more used to seeing it for multiple seconds (2-3 sec) until it loads. I observe the same anecdotal speedup for other sites... drive, youtube, calendar. I wonder if they are throwing all the hardware they have at their serv…
If this phenomenon is actually real instead of just perception then I'd guess it is down to reduced demand of some short. Some possibilities:
a) users haven't all come back yet b) Google is throttling how fast users can access services again to prevent further outages c) to reduce load, apps have features turned off (which might make things directly faster on the user's end or just reduce load on the server side)
Re: Google outage – resolved
#658Earlier quoted context omitted.
> I do not envy Google engineers right now. A few years ago I released a bug in production that prevented users from logging into our desktop app. It affected about ~1k users before we found out and rolled back the release. I still remember a very cold feeling in my belly, barely could sleep that night. It is difficult to imagine what the people responsible for this are feeling right now.
If you alone were able to do it, then the system was designed badly. The bigger the impact, the more robust it has to be to prevent accidents.
[deleted]
Re: Google outage – resolved
#659Google Home and Chromecast also rendered useless. _Hey Google, how are you?_ leads to some very passive aggressive replies.
"Hey Google!"
"Something went wrong. Try again in a few seconds."
Re: Google outage – resolved
#6604:41AM PT, Google services have been restored to my accounts (free & gsuite). And I have never seen them load so fast before - gmail progress bar barely seen for a fraction of a second whereas I am more used to seeing it for multiple seconds (2-3 sec) until it loads. I observe the same anecdotal speedup for other sites... drive, youtube, calendar. I wonder if they are throwing all the hardware they have at their serv…
I wish it was always like this. I hate how slow YouTube, Gmail, etc. often are normally.