Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Google outage – resolved

news.ycombinator.com

Post reply on HN
Print view View original

661–670 of 870 posts

Re: Google outage – resolved

#661

Post by ithkuil »

rubatuga wrote: post #554
hackernews1134 wrote: post #535

4:41AM PT, Google services have been restored to my accounts (free & gsuite). And I have never seen them load so fast before - gmail progress bar barely seen for a fraction of a second whereas I am more used to seeing it for multiple seconds (2-3 sec) until it loads. I observe the same anecdotal speedup for other sites... drive, youtube, calendar. I wonder if they are throwing all the hardware they have at their serv…

Oh man, you're right. Bloated gmail loaded instantly. What's going on? It's loading almost 2x to 3x faster.

Isn't this a good indication that the performance problem if gmail may not be related to the "bloat" of the frontend itself?

Re: Google outage – resolved

#662

Post by tomschwiha »

oAlbe wrote: post #424
jacquesm wrote: post #196

So, anybody still feel like arguing that 'the cloud' is a viable back-up? Or is that a sore point right now? Just for a moment imagine: what if it never comes back again? Of course it will, - at least, it better - but what if it doesn't? And if it does, are you going to take countermeasures in case it happens again or is it just going to be 'back to normal' again?

What worries me the most is email. I basically don't use any other Google services other than Gmail and YouTube, but for email I really don't know of an alternative. Sure you can argue "move to Fastmail/Protonmail/Hey/whatever", but those can also go down on you just like Google is down now. And self hosting email is apparently not a thing due to complexity and having to forever fight with being marked as spam (ndr.:…

I've been happy with using Hosted Exchange on Microsoft. I own the domain so ultimately I can point the DNS to some other provider. Outlook stores the mails locally so I have a backup. I think the most important thing about E-Mail is to receive future E-Mails and not look at historical ones. In the end you can always ask the receipent to send you a copy of the email conversation - if you dont own the domain it get much harder to convince you actually own the email.

Re: Google outage – resolved

#663

Post by matkoniecz »

jacquesm wrote: post #196

So, anybody still feel like arguing that 'the cloud' is a viable back-up? Or is that a sore point right now? Just for a moment imagine: what if it never comes back again? Of course it will, - at least, it better - but what if it doesn't? And if it does, are you going to take countermeasures in case it happens again or is it just going to be 'back to normal' again?

Of course cloud is a viable back up, similarly to physical drives.

Both have vastly different failure modes and typical backup should use both of them.

This way if all my backups are gone I likely have way more important issues that loss of files.

(and yes, my backups are encrypted)

Re: Google outage – resolved

#665

Post by a3_nm »

rplnt wrote: post #641
tarruda wrote: post #529

Earlier quoted context omitted.

> I do not envy Google engineers right now. A few years ago I released a bug in production that prevented users from logging into our desktop app. It affected about ~1k users before we found out and rolled back the release. I still remember a very cold feeling in my belly, barely could sleep that night. It is difficult to imagine what the people responsible for this are feeling right now.

If you alone were able to do it, then the system was designed badly. The bigger the impact, the more robust it has to be to prevent accidents.

The big mistake in the system is that everyone in the world is relying on Google services... These problems would have less impact with a more diverse ecosystem.

Re: Google outage – resolved

#666

Post by ilikehurdles »

mmillin wrote: post #121

Given the blast radius of this (all regions appear to be impacted) along with the fact that services that don't rely on auth are working as normal, it must be a global authN/Z issue. I do not envy Google engineers right now.

Coincidentally, Google Authenticator was finally just updated on iOS after many years without update.

Re: Google outage – resolved

#667

Post by igetspam »

oAlbe wrote: post #609
beshrkayali wrote: post #508

Earlier quoted context omitted.

The point is that you shouldn't put all your eggs in one basket. All services go down. If you're worried about someone else handling it when it goes down then host your own [1], otherwise you can use something different for each thing you need. Don't rely on Google for everything. 1- https://github.com/awesome-selfhosted/awesome-selfhosted#ema...

Yes, I know what's the point. But how do you avoid putting all your eggs in one basket? You can't host your email on more than one "provider" (including self host), and the vast majority of important services that you link your email to (bank, digital identity, taxes and other government services) does not allow you to have more than one linked to it; which means, that one goes down, you don't have one. Sure, I can g…

> You can't host your email on more than one "provider"

You can do split delivery and have your email be delivered to two different destinations. It's less common than it used to be but it's trivial.

Re: Google outage – resolved

#668

Post by ignoramous »

papito wrote: post #642
ignoramous wrote: post #596

Earlier quoted context omitted.

Same. At AWS, I once took an entire AZ down of a public-facing production service (with a mis-typed command), but that was nothing compared to when I accidentally deleted an entire region via internal console (too many browser tabs). Thank goodness turned out to be unused / unlaunched, non-production stack. I felt horrible for hours despite zero impact (in both the cases).

Jesus. One would think you'd have some safeguards for that. Even Dropbox will give you an alert if you try to nuke over 1,000 files. More reasons to COLOR CODE your work environments, if possible.

Yes but that was eons ago. The safeguards are well and truly in-place now. Not just one, several in fact.

Re: Google outage – resolved

#669

Post by eric_khun »

If you pay for Google Services, they have an SLA (service level agreement) of 99.9% [1]. If their services are down more than 43 minutes this month[2], you can request “free days” of usage.

Edit: Services were down from ~12:55pm to ~1:52pm, it's 57minutes. Thanks hiby007

[1] https://workspace.google.com/intl/en/terms/sla.html

[2] https://en.wikipedia.org/wiki/High_availability#Percentage_c...

Re: Google outage – resolved

#670

Post by archi42 »

burnthrow wrote: post #653
kabes wrote: post #622

When I tried to login it said: 'this account does not exist'. So my first thought was some algorithm made a mistake and my account got deleted for no reason. I already imagined the only solution now was to write a medium post and hope it gets some traction on hackernews and google support steps in. Thinking to myself I was an idiot for knowing all this and still thinking it wouldn't happen to me. And even though it t…

Those hosting with their own domains on Google mail had an identical experience, if it makes you feel differently.

The advantage of a custom domain name is: As long as you can update DNS records, you can have your mail easily hosted elsewhere.

Obviously not relevant for this kind of outage, but in the scenario outlined by GP - Google randomly kills you off, and there is nothing you can do - this is at least an emergency strategy.

Post reply on HN