Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Google outage – resolved

news.ycombinator.com

Post reply on HN
Print view View original

741–750 of 870 posts

Re: Google outage – resolved

#741

Post by gravitas »

boringg wrote: post #727
nickjj wrote: post #719

Earlier quoted context omitted.

This topic just came up recently on a podcast I was on where someone said a large service was down for X amount of time and the service being down tanked his entire business while it was down for days. But he was compensated in hosting credits for the exact amount of down time for the 1 service that caused the issue. It took so long to resolve because it took support a while to figure out it was their service, not hi…

Completely agree with your analogy but have you ever seen any SLA that provides any additional liability? I haven't seen them - you are stuck either relying on hosting services SLA or DIY.

I cannot share details for the obvious reason, but yes - there are SLAs signed into contract directly behind the scenes which result in automatic payouts of a condition isn't met and it's not a simple credit.

Enterprise level SLAs are crafted by lawyers in negotiations behind the scenes and are not the same as what you see on random public services. Our customers have them with us, and we have them with our vendors. Contract negotiations take months at the $$$$ level.

Re: Google outage – resolved

#742

Post by itsrajju »

ants_a wrote: post #720
tzs wrote: post #688

Earlier quoted context omitted.

You are assuming that a minute of disruption can not cause more than a minute's loss of productivity. I don't think that assumption is justified. Consider an exactly one minute outage that affects multiple things I use for work. First, I may not immediately recognize that the outage is actually with some single service provider. If several things are out I'm probably going to suspect it is something on my end, or may…

I'm also assuming most of the world doesn't grind to a halt when gmail is down. Crops keep growing and factories keep running.

Even software engineers who are in a state of flow keep working :)

Re: Google outage – resolved

#743

Post by Sodman »

jellykid wrote: post #729
nickjj wrote: post #719

Earlier quoted context omitted.

This topic just came up recently on a podcast I was on where someone said a large service was down for X amount of time and the service being down tanked his entire business while it was down for days. But he was compensated in hosting credits for the exact amount of down time for the 1 service that caused the issue. It took so long to resolve because it took support a while to figure out it was their service, not hi…

I've never seen an SLA that compensates for anything more than credit off your bill. I can't imagine a service that pays for loss of productivity, one outage and the whole company could be bankrupt. If your business depends on a cloud service for productivity you should have a backup plan if that service goes down.

I haven't seen one (at least for a SaaS company) that will compensate for loss of productivity/revenue etc, but something like Slack's SLA[0] seems like it's moving in the right direction. They guarantee a 99.99% uptime (max downtime of 4 min/22 seconds per month) and give 10x credits for any downtime.

Granted, there's probably not many businesses that are losing major revenue because slack's down for half an hour, but it's nice to at least see them acknowledge that 1 minute down deserves more than 1 minute of refunds!

[0] https://slack.com/terms/service-level-agreement

Re: Google outage – resolved

#744

Post by Maxion »

huijzer wrote: post #716
xenocratus wrote: post #703

Earlier quoted context omitted.

I'm reminded of the quote from Thomas J. Watson: > Recently, I was asked if I was going to fire an employee who made a mistake that cost the company $600,000. No, I replied, I just spent $600,000 training him. Why would I want somebody to hire his experience?

Agreed, and also it's worth noting that we're talking about companies here. Yes, for any individual the amount of money lost is insane, but that's the risk for the company. If one individual can accidentally nearly bankrupt the company, then the company did not have proper risk management in place. That isn't too say that it wouldn't also affect my sleep quality.

Sadly a lot of managers don't see it this way, they'd rather assign blame.

Re: Google outage – resolved

#745

Post by sn41 »

rightbyte wrote: post #648
sn41 wrote: post #616

Earlier quoted context omitted.

Underrated comment. If the government freezes your bank account, at least you can take them to court. If Google mistakenly disables your account, there does not seem to be any legal recourse. Considering the extent to which people depend on their services, perhaps there should be a more elaborate appeal process.

Google doesn't disable ordinary rulefollowing accounts by mistake. They are aware of the false positive rate and don't care.

I agree, but the problem is that "ordinary rulefollowing" is not clearly defined by Google. Perhaps it was that innocent park frisbee party video with a Metallica sound played in the background by some other party there... and not only your video, but also your YouTube account, and worse, email account gets blocked. Maybe this scenario is dystopian - the point is that it is a black-box no-appeal/limited appeal system when such an event happens.

Re: Google outage – resolved

#746

Post by waisbrot »

patrickthebold wrote: post #736
nickjj wrote: post #719

Earlier quoted context omitted.

This topic just came up recently on a podcast I was on where someone said a large service was down for X amount of time and the service being down tanked his entire business while it was down for days. But he was compensated in hosting credits for the exact amount of down time for the 1 service that caused the issue. It took so long to resolve because it took support a while to figure out it was their service, not hi…

Seems like you want insurance. As with the hospital bill you'd generally be paying a bunch of extra money for your health insurance plan to not get stuck with the bill. Not sure that exists for businesses, but I'd expect you'd need to go shopping separately if you want that. Seems like a good business idea if it doesn't exist.

I think the idea here is that if the payment for SLA breach is just "don't pay for the time we were down" or (as I've seen in other SLAs) "half off during the time we were down" that doesn't feel like much of an incentive on the service provider.

They have other incentives, obviously, like if everyone talks about how Google is down then that's bad for future business. But when thinking of SLAs I'm always surprised when they're not more drastic. Like "over 0.1% downtime: free service for a month".

Re: Google outage – resolved

#747

Post by adwww »

alias_neo wrote: post #707
shawkinaw wrote: post #692

Earlier quoted context omitted.

Engineers shouldn’t deploy to prod directly, but sometimes it’s necessary to SSH into an instance for logs, stack dumps, etc. Source: worked for 2 big to very big tech cos.

For a large or v large tech co you should probably be aggregating logs to a centralised location that doesn't require access to production systems in this way. Stack dumps should also be collected safely off-system if necessary. Perhaps my industry is a little more security conscious (I don't know which industry you're talking about), but this doesn't seem like good practice.

Who fixes the centralised log system when that needs debugging?

Unless prohibited in something like banking, following best practice to the letter is sometimes unacceptably slow for most industries.

Re: Google outage – resolved

#748

Post by shawkinaw »

alias_neo wrote: post #707
shawkinaw wrote: post #692

Earlier quoted context omitted.

Engineers shouldn’t deploy to prod directly, but sometimes it’s necessary to SSH into an instance for logs, stack dumps, etc. Source: worked for 2 big to very big tech cos.

For a large or v large tech co you should probably be aggregating logs to a centralised location that doesn't require access to production systems in this way. Stack dumps should also be collected safely off-system if necessary. Perhaps my industry is a little more security conscious (I don't know which industry you're talking about), but this doesn't seem like good practice.

Let me be clear, I agree it should not be normal to SSH into a prod box. Our logs are centrally aggregated. But it’s one thing to say it’s not normal, but quite another to say engineers shouldn't have access, because I totally disagree with that.

Re: Google outage – resolved

#749

Post by ardy42 »

ignoramous wrote: post #596
tarruda wrote: post #529

Earlier quoted context omitted.

> I do not envy Google engineers right now. A few years ago I released a bug in production that prevented users from logging into our desktop app. It affected about ~1k users before we found out and rolled back the release. I still remember a very cold feeling in my belly, barely could sleep that night. It is difficult to imagine what the people responsible for this are feeling right now.

Same. At AWS, I once took an entire AZ down of a public-facing production service (with a mis-typed command), but that was nothing compared to when I accidentally deleted an entire region via internal console (too many browser tabs). Thank goodness turned out to be unused / unlaunched, non-production stack. I felt horrible for hours despite zero impact (in both the cases).

> At AWS, I once took an entire AZ down of a public-facing production service (with a mis-typed command), but that was nothing compared to when I accidentally deleted an entire region via internal console (too many browser tabs). Thank goodness turned out to be unused / unlaunched, non-production stack. I felt horrible for hours despite zero impact (in both the cases).

It seems like a design flaw for actions like that to be so easy. E.g.

> Hey, we detected you want to delete an AWS region. Please have an authorized coworker enter their credentials to second your command.

Re: Google outage – resolved

#750

Post by Cthulhu_ »

gsich wrote: post #684
kaushikt wrote: post #676

Earlier quoted context omitted.

why?

In this example: you get free days. Which depending on your business might be worthless if you have suffered more monetary loss due to the downtime than the free days are worth.

Exactly; downtime doesn't cost a cloud service much. At worst it causes reputation damage, with possibly large companies deciding to go for a competitor, losing a contract worth tens or hundreds of millions.
Post reply on HN