Google outage – resolved • Hacker News Board

Re: Google outage – resolved

#641

Post by rplnt » Mon, Dec 14, 2020, 1:18 PM UTC

tarruda wrote: post #529

mmillin wrote: post #121

Given the blast radius of this (all regions appear to be impacted) along with the fact that services that don't rely on auth are working as normal, it must be a global authN/Z issue. I do not envy Google engineers right now.

> I do not envy Google engineers right now. A few years ago I released a bug in production that prevented users from logging into our desktop app. It affected about ~1k users before we found out and rolled back the release. I still remember a very cold feeling in my belly, barely could sleep that night. It is difficult to imagine what the people responsible for this are feeling right now.

If you alone were able to do it, then the system was designed badly. The bigger the impact, the more robust it has to be to prevent accidents.

Re: Google outage – resolved

#642

Post by papito » Mon, Dec 14, 2020, 1:18 PM UTC

ignoramous wrote: post #596

tarruda wrote: post #529

Earlier quoted context omitted.

> I do not envy Google engineers right now. A few years ago I released a bug in production that prevented users from logging into our desktop app. It affected about ~1k users before we found out and rolled back the release. I still remember a very cold feeling in my belly, barely could sleep that night. It is difficult to imagine what the people responsible for this are feeling right now.

Same. At AWS, I once took an entire AZ down of a public-facing production service (with a mis-typed command), but that was nothing compared to when I accidentally deleted an entire region via internal console (too many browser tabs). Thank goodness turned out to be unused / unlaunched, non-production stack. I felt horrible for hours despite zero impact (in both the cases).

Jesus. One would think you'd have some safeguards for that. Even Dropbox will give you an alert if you try to nuke over 1,000 files. More reasons to COLOR CODE your work environments, if possible.

Re: Google outage – resolved

#643

Post by shawabawa3 » Mon, Dec 14, 2020, 1:18 PM UTC

Mistri wrote: post #182

It's super interesting that all Google services that I've tried are down _except_ for Google Search. What would isolate Search from the rest of Google's products such that it wouldn't be affected by a mass outage like this?

Google search was slow for me during the outage (5-10s per request)

I imagine google search only has optional dependencies on login and correctly falls-back if these fail

Re: Google outage – resolved

#644

Post by roadnottaken » Mon, Dec 14, 2020, 1:19 PM UTC

hackernews1134 wrote: post #535

4:41AM PT, Google services have been restored to my accounts (free & gsuite). And I have never seen them load so fast before - gmail progress bar barely seen for a fraction of a second whereas I am more used to seeing it for multiple seconds (2-3 sec) until it loads. I observe the same anecdotal speedup for other sites... drive, youtube, calendar. I wonder if they are throwing all the hardware they have at their serv…

I wonder if they just killed the affected service so it's loading faster-than-usual now

Re: Google outage – resolved

#645

Post by unixhero » Mon, Dec 14, 2020, 1:19 PM UTC

The appstatus page from Google is a good indicator for me of which products are currently live from Google. Uhm, more of a sidenote :)

Re: Google outage – resolved

#646

Post by papito » Mon, Dec 14, 2020, 1:19 PM UTC

tarruda wrote: post #529

mmillin wrote: post #121

Given the blast radius of this (all regions appear to be impacted) along with the fact that services that don't rely on auth are working as normal, it must be a global authN/Z issue. I do not envy Google engineers right now.

> I do not envy Google engineers right now. A few years ago I released a bug in production that prevented users from logging into our desktop app. It affected about ~1k users before we found out and rolled back the release. I still remember a very cold feeling in my belly, barely could sleep that night. It is difficult to imagine what the people responsible for this are feeling right now.

When I was interviewing at Morgan Stanley, I asked "how do you do this job if a mistake can cost people money?".

The answer was "well, if you don't do anything, you make NO money".

Re: Google outage – resolved

#647

Post by q3k » Mon, Dec 14, 2020, 1:20 PM UTC

hackernews1134 wrote: post #535

4:41AM PT, Google services have been restored to my accounts (free & gsuite). And I have never seen them load so fast before - gmail progress bar barely seen for a fraction of a second whereas I am more used to seeing it for multiple seconds (2-3 sec) until it loads. I observe the same anecdotal speedup for other sites... drive, youtube, calendar. I wonder if they are throwing all the hardware they have at their serv…

I would guess autoscaling kicked in (RPC error rate caused higher CPU usage?) and now things will scale back down again.

Re: Google outage – resolved

#648

Post by rightbyte » Mon, Dec 14, 2020, 1:21 PM UTC

sn41 wrote: post #616

cvrjk wrote: post #304

Earlier quoted context omitted.

For a second I legit thought Google banned my account for some reason. And I don't like it that I feel relieved..

Underrated comment. If the government freezes your bank account, at least you can take them to court. If Google mistakenly disables your account, there does not seem to be any legal recourse. Considering the extent to which people depend on their services, perhaps there should be a more elaborate appeal process.

Google doesn't disable ordinary rulefollowing accounts by mistake. They are aware of the false positive rate and don't care.

Re: Google outage – resolved

#649

Post by jerrygreenest » Mon, Dec 14, 2020, 1:22 PM UTC

Not just google, hackernews was also displaying: "We're having some trouble serving your request. Sorry!", has it something to do with google architecture?

Btw, that's funny you listed all the google services exactly in the order I found them unavailable: first youtube, then gmail, then docs

Re: Google outage – resolved

#650

Post by PeterStuer » Mon, Dec 14, 2020, 1:23 PM UTC

mrcus wrote: post #292

mmillin wrote: post #121

Given the blast radius of this (all regions appear to be impacted) along with the fact that services that don't rely on auth are working as normal, it must be a global authN/Z issue. I do not envy Google engineers right now.

Nope, especially considering the implications of this, with the amount of people working remotely. Google Meet, Classroom, etc. are down. This is probably literally costing billions every minute just in loss of productivity.

Besides the exaggerated figure, I always find these claims bizarre. Sure, there was some momentary loss, but aggregated over a month this will not even register.