Your nines are not my nines • Hacker News Board

Re: Your nines are not my nines

#101

Post by zaptheimpaler » Wed, Jul 17, 2019, 7:03 AM UTC

windexh8er wrote: post #78

m463 wrote: post #32

Earlier quoted context omitted.

I worked at a company once where each bug had a really interesting field: root cause I wish I could remember the values you could fill in, they were very intelligently chosen. What I learned: if you didn't know what the root cause was, you probably didn't fix anything.

1000% this. I've been in many orgs where root cause was either completely missing or completely missed the point. Recently I quit a company that thought they were doing RCA. The fact they sent out an email that there was an outage and then N number of hours or days later another email to say it was "fixed" and that they scaled or thought they found the problem and don't worry about it anymore. Literally we had weeks…

So how did the leadership respond to your questions? Did they agree, and did things change?

Re: Your nines are not my nines

#102

Post by titzer » Wed, Jul 17, 2019, 7:30 AM UTC

dragonwriter wrote: post #94

vincentmarle wrote: post #65

Earlier quoted context omitted.

> if 10 of your cloud services each have a reliability of 99.999, all together the reliability is not 99.999. (The answer is 99.99)

It's probably not in practice, since that assumes failures are perfectly independent. If they are perfectly correlated, the answer is still 99.999. For most real cases, it will be between those extremes.

I think it depends on how you define availability.

1. Suppose we define availability as "at least one is up". If the failures are completely independent, then the probability of any one being down is 10^-5 (five nines) and the probability of all 10 being down at the same time is (10^-5)^10 = 10^-50 (fifty nines).

2. If we instead define availability as "all 10 are up" (which is essentially equivalent to one failure causes a cascading failure) then in the same scenario where failures are independent, this is (1-10^-5)^10 ~= 99.99% (four nines).

Re: Your nines are not my nines

#103

Post by sumanthvepa » Wed, Jul 17, 2019, 7:47 AM UTC

I usually advise my clients to treat cloud providers like they would treat hardware, perhaps a bit more reliable (though in practice that is usually not the case.) If you cannot afford for your database to unavailable, invest in creating a backup db for your database. That advice holds regardless of whether said database is run on a cloud or not. Base your investment decisions on the downtime you observe of the cloud provider. Don't expect to make your cloud provider magically able to ensure your app has n 9s reliability. That's on you.

Re: Your nines are not my nines

#104

Post by AstralStorm » Wed, Jul 17, 2019, 8:56 AM UTC

tschellenbach wrote: post #66

I think these blogposts create a false narrative. It should start by acknowledging that in 99% of cases any outsourced systems' stability will be better than anything in-house. Yes there are some vendors that don't do a good job. I heard many people complaining about Layer for instance. But most vendors, AWS, Stripe, Algolia, Stream, can invest more time and effort in stability than you can feasible do for an in-hous…

Will it? The response times to failure matter more oftentimes than the availability itself. Redundancy as well, which can be cheaper manually handled than relying on some unknown process of a provider. The support on any cloud is terrible for a small client.

Re: Your nines are not my nines

#105

Post by s_Hogg » Wed, Jul 17, 2019, 11:33 AM UTC

Annatar wrote: post #98

hinkley wrote: post #26

Earlier quoted context omitted.

Maybe it's time for a consumer watchdog group to step in and do their own reporting for services like this. Like https://www.isitdownrightnow.com/ but with sharper teeth. I'm not sure how much time I have to participate but I wouldn't mind chipping in a bit on a co-op in this space. But it might be easier to convince Is it Down Right Now to grow some fangs, or socialize the idea that it does (perception counts for a…

Or it's time to go back to one's own infrastructure and take one's own destiny into one's own hands, along with the responsibility.

I can't remember ever seeing this work out well lol. Happy to be proven wrong one day, though.

Re: Your nines are not my nines

#106

Post by badpun » Wed, Jul 17, 2019, 11:58 AM UTC

jaytaylor wrote: post #90

rossjudson wrote: post #88

Earlier quoted context omitted.

You might want to have a look at some Google Cloud SLAs[1]. They are generally calculated based on actual performance on a particular customer's RPC traffic, down to individual RPCs (at least on Google Cloud Storage, where I work). Read through the agreements to the definition of "Error Rate", and I think you'll find the terms you're looking for. You're welcome! :) The root post does raise an important issue, though…

How are Google Cloud SLAs valuable when the service regularly has multi-hour (IIRC, > 5 hours!) (sometimes global) outages? I pose that Google is not presently in a good position to highlight as a role model or case-study for demonstration of effective cloud provider SLAs.

>How are Google Cloud SLAs valuable when the service regularly has multi-hour (IIRC, > 5 hours!) (sometimes global) outages?

You can sue them? (if that's not forbidden by the contract).

Re: Your nines are not my nines

#107

Post by darkcha0s » Wed, Jul 17, 2019, 12:15 PM UTC

JTbane wrote: post #40

Even funnier, when a massive network outage occurs, cloud providers shrug and say "not my problem :-)"

How would you mitigate that if you were running it on-premise?

Re: Your nines are not my nines

#108

Post by darkcha0s » Wed, Jul 17, 2019, 12:19 PM UTC

altmind wrote: post #15

ex3xu wrote: post #8

Isn't this why service-level agreements exist? If the nines of uptime are that important to your business and you don't want to be a gnat on a windshield, you've got to give the vendor some financial incentive to pay attention to you, right? Or is Rachel talking about a situation where you have an SLA in place, but you can't even prove downtime to the vendor because their monitoring software is inadequate?

To get a proper SLA you need to pay for SLA. What SLA google, azure, aws have is useless, some service credit proportional to the outage duration. Totally nothing to cover lost profits or direct damage. For the last gcloud outage, i think you have to talk to people and APPLY for a credit, obviously very few did that https://news.ycombinator.com/item?id=20078296

Working with enterprises that actually have individual SLA's with one of the clouds you mentioned, this is not true. You can handle out your own SLA's with certain providers, and not just get "credits." These enterprises have mature enough monitoring solutions to be able to prove to the provider that they didn't hold up on their end. Besides that, every half-way capable solution architect wouldn't move "system critical" software to the cloud, hence reducing "lost profits" or "direct damage". If you're doing that, it's seriously your own fault.

Re: Your nines are not my nines

#109

Post by jameshart » Wed, Jul 17, 2019, 12:39 PM UTC

bosie wrote: post #76

jameshart wrote: post #61

Earlier quoted context omitted.

The issue is that percentiles are a very crude tool. This isn’t just a multitenancy problem, it can manifest in any multiuser system. If 0.5% of your traffic comes from New Zealand, a DNS issue affecting your CDN routing that causes all NZ traffic to time out won’t affect your 99th percentile loadtime graph at all. Essentially, percentiles are useless for discovering problems that have a strong effect on a small port…

What scatter plot and/or histogram do you have in mind to dig up the small but consistent group in the outlier group?

Ah, yeah - figuring out what they have in common is the trick :)

Good APMs and trace tools will let you zero in on traces by characteristics - so if you notice there’s a bump in requests which have a 2 second load time, you can select them all and analyze how they are distributed - whether they are mostly one browser, one location, or one user even. But you need a solid strategy for tagging and logging traces.

Re: Your nines are not my nines

#110

Post by kwhitefoot » Wed, Jul 17, 2019, 1:21 PM UTC

Surely when you use buy such a service the uptime guarantee applies to the service you have paid for? What happens to the rest of the customers is irrelevant. And the average availability the aggregate is even more so.