Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Google outage – resolved

news.ycombinator.com

Post reply on HN
Print view View original

711–720 of 870 posts

Re: Google outage – resolved

#711

Post by b0afc375b5 »

cvrjk wrote: post #698
papito wrote: post #646

Earlier quoted context omitted.

When I was interviewing at Morgan Stanley, I asked "how do you do this job if a mistake can cost people money?". The answer was "well, if you don't do anything, you make NO money".

Welp, as a new grad there, I had brought down one very important database server on a Sunday night (a series of really unfortunate events). Multiple senior DBAs had to be involved to resuscitate it. It started functioning normally just a few hours before market open in HK. If it was any later, it would have been some serious monetary loss. Needless to say, I was sweating bullets. Couldn't eat anything the entire day…

I'm so glad I'm not the only one feeling deployment anxiety. The project I'm involved in doesn't really have serious money involved, but when there's a regression found only after production deployment my stress levels go up a notch.

Re: Google outage – resolved

#712

Post by abluecloud »

gsich wrote: post #693
fullstop wrote: post #691

Earlier quoted context omitted.

Not OP, but how do you measure them? Let's say, for example, you can send and receive email, but attaching files does not work. Is the service up or down? What if the majority of your users can access the service, but one of your BGP peers is not routing properly and some of your users are unable to access?

Down because email attachments are base64 encoded files written in plaintext into the body. So if those are not working, email itself is not working.

that was a bad example. i guess the comment was trying to say "how do you account for partial service degradations".

(i dont think SLAs are BS btw)

Re: Google outage – resolved

#713

Post by rptr »

ayoubElk wrote: post #697
ignoramous wrote: post #596

Earlier quoted context omitted.

Same. At AWS, I once took an entire AZ down of a public-facing production service (with a mis-typed command), but that was nothing compared to when I accidentally deleted an entire region via internal console (too many browser tabs). Thank goodness turned out to be unused / unlaunched, non-production stack. I felt horrible for hours despite zero impact (in both the cases).

Irrelevant to the discussion, but I just wanted to say thank you for the categorized list of users I can follow on your profile!

wow, that list is crazy. thanks OP.

Re: Google outage – resolved

#714

Post by [deleted] »

derwiki wrote: post #696
vineyardmike wrote: post #370

Help! My Waymo taxi won't open the doors without me logging into the app. Its driving around in circles on route 500 and won't stop. /s - for now ;)

I know you’re joking, but I am curious if Waymo’s fleet was affected. Shouldn’t be right? But I’m also surprised every time I fresh login to Gmail and YouTube shows up in the intermediary redirect chain.

[deleted]

Re: Google outage – resolved

#715

Post by pas »

fullstop wrote: post #691
kaushikt wrote: post #676

Earlier quoted context omitted.

why?

Not OP, but how do you measure them? Let's say, for example, you can send and receive email, but attaching files does not work. Is the service up or down? What if the majority of your users can access the service, but one of your BGP peers is not routing properly and some of your users are unable to access?

The SLA itself should specify the way availability is measured.

Re: Google outage – resolved

#716

Post by huijzer »

xenocratus wrote: post #703
papito wrote: post #646

Earlier quoted context omitted.

When I was interviewing at Morgan Stanley, I asked "how do you do this job if a mistake can cost people money?". The answer was "well, if you don't do anything, you make NO money".

I'm reminded of the quote from Thomas J. Watson: > Recently, I was asked if I was going to fire an employee who made a mistake that cost the company $600,000. No, I replied, I just spent $600,000 training him. Why would I want somebody to hire his experience?

Agreed, and also it's worth noting that we're talking about companies here. Yes, for any individual the amount of money lost is insane, but that's the risk for the company. If one individual can accidentally nearly bankrupt the company, then the company did not have proper risk management in place.

That isn't too say that it wouldn't also affect my sleep quality.

Re: Google outage – resolved

#718

Post by onychomys »

papito wrote: post #646
tarruda wrote: post #529

Earlier quoted context omitted.

> I do not envy Google engineers right now. A few years ago I released a bug in production that prevented users from logging into our desktop app. It affected about ~1k users before we found out and rolled back the release. I still remember a very cold feeling in my belly, barely could sleep that night. It is difficult to imagine what the people responsible for this are feeling right now.

When I was interviewing at Morgan Stanley, I asked "how do you do this job if a mistake can cost people money?". The answer was "well, if you don't do anything, you make NO money".

I work for an extremely famous hospital in the American midwest. We're divided into three sections, one for clinical work, one for research, and one for education. I always tell people that I'm pretty content being in research (which is less sexy than clinical), because if I screw something up, some PI's project takes ten months and one week instead of ten months. In clinical, if you screw something up, somebody dies! I just don't think I could handle that level of stress.

Re: Google outage – resolved

#719

Post by nickjj »

kaushikt wrote: post #676
api wrote: post #674

Earlier quoted context omitted.

SLAs are largely bullshit.

why?

This topic just came up recently on a podcast I was on where someone said a large service was down for X amount of time and the service being down tanked his entire business while it was down for days. But he was compensated in hosting credits for the exact amount of down time for the 1 service that caused the issue. It took so long to resolve because it took support a while to figure out it was their service, not his site.

So then I jokingly responded with that being like going to a restaurant, getting massive food poisoning, almost dying, ending up with a $150,000 hospital bill and then the restaurant emails you with "Dear valued customer, we're sorry for the inconvenience and have decided to award you a $50 gift card for any of our restaurants, thanks!".

If your SLA agreement is only for precisely calculated credits, that's not really going to help in the grand scheme of things.

Re: Google outage – resolved

#720

Post by ants_a »

tzs wrote: post #688
ants_a wrote: post #519

Earlier quoted context omitted.

Total world economic output is ~$150M / minute, so billions every minute is off by few orders of magnitude.

You are assuming that a minute of disruption can not cause more than a minute's loss of productivity. I don't think that assumption is justified. Consider an exactly one minute outage that affects multiple things I use for work. First, I may not immediately recognize that the outage is actually with some single service provider. If several things are out I'm probably going to suspect it is something on my end, or may…

I'm also assuming most of the world doesn't grind to a halt when gmail is down. Crops keep growing and factories keep running.
Post reply on HN