Live data from Hacker News

Google outage – resolved

news.ycombinator.com

721–730 of 870 posts

Re: Google outage – resolved

#721
post #681

Earlier quoted context omitted.

Yes but that was eons ago. The safeguards are well and truly in-place now. Not just one, several in fact.

Apart from the ones that they haven't worked out yet :)

When I meet the engineer who can design for the unknown unknowns, I will bow to them.

Re: Google outage – resolved

#722
post #594

Earlier quoted context omitted.

I guess a lot of people are fine with the risk. Everybody uses it, so if, like, Gmail loses all the emails, we are then in such a state that the consequences will be more bearable and socially normal. Most people are fine with accepting that whatever future thing will happen to most people will also happen to them. Because then the consequences will also be normal. If the apocalypse comes, it comes for almost all of…

This sounds like the good old 1970-80s "No one ever got fired for buying IBM" argument.

Still haven't heard of anyone being fired for buying IBM. Have you?

Re: Google outage – resolved

#723

If you pay for Google Services, they have an SLA (service level agreement) of 99.9% [1]. If their services are down more than 43 minutes this month[2], you can request “free days” of usage. Edit: Services were down from ~12:55pm to ~1:52pm, it's 57minutes. Thanks hiby007 [1] https://workspace.google.com/intl/en/terms/sla.html [2] https://en.wikipedia.org/wiki/High_availability#Percentage_c...

In [1] it says: "Customer Must Request Service Credit." Do you know how to request it?

Re: Google outage – resolved

#724

Earlier quoted context omitted.

why?

Not OP, but how do you measure them? Let's say, for example, you can send and receive email, but attaching files does not work. Is the service up or down? What if the majority of your users can access the service, but one of your BGP peers is not routing properly and some of your users are unable to access?

Google do a very good job of defining their SLAs.

In answer to your question, they'll accept evidence from your own monitoring system when you claim on the SLA. They pair that up to their own knowledge about how the system was performing, then make the grant.

Google are exceptionally good at this, from my experience. Far better than most other companies, who aim to provide as little detail as possible while getting away with 'providing an SLA'.

Re: Google outage – resolved

#725
post #698

Earlier quoted context omitted.

Welp, as a new grad there, I had brought down one very important database server on a Sunday night (a series of really unfortunate events). Multiple senior DBAs had to be involved to resuscitate it. It started functioning normally just a few hours before market open in HK. If it was any later, it would have been some serious monetary loss. Needless to say, I was sweating bullets. Couldn't eat anything the entire day…

I'm so glad I'm not the only one feeling deployment anxiety. The project I'm involved in doesn't really have serious money involved, but when there's a regression found only after production deployment my stress levels go up a notch.

When I was working at a pretty big IT provider in the electronic banking sector, we (management and senior devs) made it an unspoken rule, that: - Juniors shall also handle production deployments regularly. - A senior person is always on call (even if only unofficially / off the clock). - Junior devs are never blamed for fuckups, irrespective of the damage they caused.

That was the only way to help people develop routine regarding big production deployments.

Re: Google outage – resolved

#726
post #424

Earlier quoted context omitted.

What worries me the most is email. I basically don't use any other Google services other than Gmail and YouTube, but for email I really don't know of an alternative. Sure you can argue "move to Fastmail/Protonmail/Hey/whatever", but those can also go down on you just like Google is down now. And self hosting email is apparently not a thing due to complexity and having to forever fight with being marked as spam (ndr.:…

I'm running my own mail server, and I think anyone who has some experience with Linux should be able to do the same in a day or two. Once it's set up it just works. You can still use Gmail and fall back to connecting directly to your server if Gmail is down. Some mails might be flagged as spam if the IP/domain has no reputation, but that quickly passes, at least that's my experience.

I specifically use Gsuite so that I don't have to deal with managing a spam filter or dealing with IP reputation issues. I'd be willing to self-host almost anything else.

Re: Google outage – resolved

#727
post #719

Earlier quoted context omitted.

why?

This topic just came up recently on a podcast I was on where someone said a large service was down for X amount of time and the service being down tanked his entire business while it was down for days. But he was compensated in hosting credits for the exact amount of down time for the 1 service that caused the issue. It took so long to resolve because it took support a while to figure out it was their service, not hi…

Completely agree with your analogy but have you ever seen any SLA that provides any additional liability? I haven't seen them - you are stuck either relying on hosting services SLA or DIY.

Re: Google outage – resolved

#729
post #719

Earlier quoted context omitted.

why?

This topic just came up recently on a podcast I was on where someone said a large service was down for X amount of time and the service being down tanked his entire business while it was down for days. But he was compensated in hosting credits for the exact amount of down time for the 1 service that caused the issue. It took so long to resolve because it took support a while to figure out it was their service, not hi…

I've never seen an SLA that compensates for anything more than credit off your bill. I can't imagine a service that pays for loss of productivity, one outage and the whole company could be bankrupt. If your business depends on a cloud service for productivity you should have a backup plan if that service goes down.

Re: Google outage – resolved

#730
post #424

So, anybody still feel like arguing that 'the cloud' is a viable back-up? Or is that a sore point right now? Just for a moment imagine: what if it never comes back again? Of course it will, - at least, it better - but what if it doesn't? And if it does, are you going to take countermeasures in case it happens again or is it just going to be 'back to normal' again?

What worries me the most is email. I basically don't use any other Google services other than Gmail and YouTube, but for email I really don't know of an alternative. Sure you can argue "move to Fastmail/Protonmail/Hey/whatever", but those can also go down on you just like Google is down now. And self hosting email is apparently not a thing due to complexity and having to forever fight with being marked as spam (ndr.:…

A lot of domain registrars will host/relay mail for you if you don't want to think about it. Otherwise it's not too hard to host yourself. The sucky part is when it breaks because you can't really just put off fixing it.
Post reply on HN