Google outage – resolved • Hacker News Board

Re: Google outage – resolved

#631

Post by wjdp » Mon, Dec 14, 2020, 1:13 PM UTC

fny wrote: post #63

tosh wrote: post #24

Just checked https://www.google.com/appsstatus all green, which does not reflect reality for me (e.g. Gmail is down) edit: shows how incredibly difficult introspection is

https://downdetector.com/ is remarkably good at catching this too. But funnily enough, a lot of the votes come from traffic that searches for "is ____ down?" on Google. XD

Found this relatively recently, seems to do a good job.

Do love how more consumer services (ISPs &c) always have some report of being down somewhere but its means nothing unless there's a big spike.

Re: Google outage – resolved

#632

Post by optimalsolver » Mon, Dec 14, 2020, 1:13 PM UTC

>drive not working

Something something someone else's computer.

Re: Google outage – resolved

#633

Post by satysin » Mon, Dec 14, 2020, 1:14 PM UTC

Waiting for the post-mortem to see it was all because someone forgot to renew an SSL cert that nobody knew was a single point of failure for literally everything.

:D

Re: Google outage – resolved

#634

Post by h43k3r » Mon, Dec 14, 2020, 1:15 PM UTC

gfitz wrote: post #336

h43k3r wrote: post #240

Everything is affected. Its Gaia(Google Auth) outage, most probably. Disclaimer - Googler here whose workplace chat is not working.

The nature of this downtime is quite severe. How are Googlers resolving this if their internal communications are down?

There is IRC as backup

Re: Google outage – resolved

#635

Post by xdavidliu » Mon, Dec 14, 2020, 1:15 PM UTC

vineyardmike wrote: post #370

Help! My Waymo taxi won't open the doors without me logging into the app. Its driving around in circles on route 500 and won't stop. /s - for now ;)

how's the converted offshore oil platform treating you?

Re: Google outage – resolved

#636

Post by JorgeGT » Mon, Dec 14, 2020, 1:16 PM UTC

mkl95 wrote: post #563

How does this incident affect IoT devices? I'd expect many of them to be using Firebase just like HN.

People were reporting not being able to turn their lights on: https://twitter.com/joemfbrown/status/1338452107419148290

Re: Google outage – resolved

#637

Post by oknoorap » Mon, Dec 14, 2020, 1:16 PM UTC

sn41 wrote: post #598

oknoorap wrote: post #523

It's time to decentralized Gmail, Youtube, and Google Docs, maybe their Search Engine too.

That's what it was 15 years back. I was slightly concerned when Google Mail and Chat were integrated. Look where we are now. Right from "The battle of the red cliffs", we will never learn the disadvantage of lashing too many boats together.

But our tech right now is far more advanced than 15 years ago.

We have IPFS, Blockchain, dat protocol today. I think it's possible to kill the giants.

Even, Tim Berners-Lee want to decentralized the web: https://techcrunch.com/2018/10/09/tim-berners-lee-is-on-a-mi...

Re: Google outage – resolved

#638

Post by domano » Mon, Dec 14, 2020, 1:16 PM UTC

tarruda wrote: post #529

mmillin wrote: post #121

Given the blast radius of this (all regions appear to be impacted) along with the fact that services that don't rely on auth are working as normal, it must be a global authN/Z issue. I do not envy Google engineers right now.

> I do not envy Google engineers right now. A few years ago I released a bug in production that prevented users from logging into our desktop app. It affected about ~1k users before we found out and rolled back the release. I still remember a very cold feeling in my belly, barely could sleep that night. It is difficult to imagine what the people responsible for this are feeling right now.

I remember how one of our engineers had his docker daemon connected to production instead of his local one and casually did a docker rm -f $(docker ps -aq) .

Same thing happened to me but with CI, which felt bad enough already.

Re: Google outage – resolved

#639

Post by tibbydudeza » Mon, Dec 14, 2020, 1:16 PM UTC

hackernews1134 wrote: post #535

4:41AM PT, Google services have been restored to my accounts (free & gsuite). And I have never seen them load so fast before - gmail progress bar barely seen for a fraction of a second whereas I am more used to seeing it for multiple seconds (2-3 sec) until it loads. I observe the same anecdotal speedup for other sites... drive, youtube, calendar. I wonder if they are throwing all the hardware they have at their serv…

Perhaps they rebooted their clusters and it flushed the memory ???.

Re: Google outage – resolved

#640

Post by cube00 » Mon, Dec 14, 2020, 1:16 PM UTC

tarruda wrote: post #529

mmillin wrote: post #121

Given the blast radius of this (all regions appear to be impacted) along with the fact that services that don't rely on auth are working as normal, it must be a global authN/Z issue. I do not envy Google engineers right now.

> I do not envy Google engineers right now. A few years ago I released a bug in production that prevented users from logging into our desktop app. It affected about ~1k users before we found out and rolled back the release. I still remember a very cold feeling in my belly, barely could sleep that night. It is difficult to imagine what the people responsible for this are feeling right now.

Don't be too hard on yourself, no dev works in a silo, there is usually user acceptance testing and product owner sign offs involved so they also have to wear some of this too.