Live data from Hacker News

Your nines are not my nines

rachelbythebay.com

111–120 of 135 posts

Re: Your nines are not my nines

#111
post #105
post #98

Earlier quoted context omitted.

Or it's time to go back to one's own infrastructure and take one's own destiny into one's own hands, along with the responsibility.

I can't remember ever seeing this work out well lol. Happy to be proven wrong one day, though.

You may have heard of a company called Amazon.

Re: Your nines are not my nines

#112
post #67

Earlier quoted context omitted.

Was it like an list of predefined values? Where I work they do root cause analysis for everything, but with freeform answers so what you describe might be different from what I'm used to. In general, I'm so used to RCA and layered mitigations (what one of our greybeards calls "belt and suspenders") that I don't know how quality happens without it. I'm a convert to the idea that if you can't fix a problem directly, th…

Unfortunately this was probably 20 years ago. I know it was a list of predefined values in a drop-down, but I'm not sure if there was a other/write-in field. The gist was that the causes were appropriate and educational. Folks couldn't choose "user is an idiot", instead having to choose "the interface was confusing".

Well "the interface was confusing" doesn't really rule out "user is an idiot", but most likely will make matters worse.

Re: Your nines are not my nines

#113
post #98
post #26

Earlier quoted context omitted.

Maybe it's time for a consumer watchdog group to step in and do their own reporting for services like this. Like https://www.isitdownrightnow.com/ but with sharper teeth. I'm not sure how much time I have to participate but I wouldn't mind chipping in a bit on a co-op in this space. But it might be easier to convince Is it Down Right Now to grow some fangs, or socialize the idea that it does (perception counts for a…

Or it's time to go back to one's own infrastructure and take one's own destiny into one's own hands, along with the responsibility.

For quite a while now I've been hoping that cloud tools will hit the turnkey solution point where anyone technically literate can manage a small private cloud. Baby pictures shouldn't be on Facebook. They should hosted on my machine, and replicated on Uncle Bill's and Aunt Sally's, and only family members have access to them.

But I think I'm going to be waiting a while.

Re: Your nines are not my nines

#115

Earlier quoted context omitted.

You may have heard of a company called Amazon.

What thread do you think you are replying to?

> own infrastructure and take one's own destiny into one's own hands, along with the responsibility.

Amazon did this. It went pretty well. So well they decided to sell the results of the expertise.

Re: Your nines are not my nines

#116
post #23

We've talked about these classes of problem from time to time, a lot of ideas have been put forward but what's the solution? If I have a multi-tenant system, and no one customer is dominant (always causes problems IMO), my 'biggest customer' might only be 4% of my traffic. There are a million things that can go wrong that make this customer's experiences different from everyone else's, from getting my sharding soluti…

The issue is that percentiles are a very crude tool. This isn’t just a multitenancy problem, it can manifest in any multiuser system. If 0.5% of your traffic comes from New Zealand, a DNS issue affecting your CDN routing that causes all NZ traffic to time out won’t affect your 99th percentile loadtime graph at all. Essentially, percentiles are useless for discovering problems that have a strong effect on a small port…

You could statistically test whether your data matches the expected distribution with automated tests.

Re: Your nines are not my nines

#117
post #55

This rings so true it hurts. At a very large, very blue, company I recall a time trying to explain to an account manager that I wanted to write the SLAs in terms of my footprint, which is to say, given the resources you have allocated to my account, lets set some SLAs like "latency from any node to any node", "latency from any node to the primary internet", "latency from any node to the secondary internet", "availabi…

Would you be willing to pay more to have the SLAs behave like that, and if so, how much more? Genuinely curious, maybe there's a market for "cloud but better SLAs".

Are there any large providers where the SLAs are meaningful, instead of a variation of "you don't have to pay us if we didn't provide the service" (i.e. something not remotely related to the damage typically caused by such outages)?

For example, Amazon will give you a 30% refund "for the individual Included Service in the affected AWS region for the monthly billing cycle in which the Unavailability occurred" if availability during a month drops to, but not below, 95% (that's a 1.5 day downtime).

That means that if your service goes 100% down because EC2 was completely broken in a region for 1.5 days, you get a refund of 9 days worth of EC2 (compute) charges, but not the associated EBS (disk) or S3 (storage) or other charges.

And "unavailability" counts only if at least two availability zones at the same time are completely down. And then you have to request the credit in a very specific format.

Google and Azure look extremely similar.

Are better SLAs typically negotiated? Because based on this, it seems like the only thing keeping cloud providers reliably above their SLAs is the fear of losing current and future customers, not the SLAs themselves. In other words, the SLAs are completely meaningless.

Re: Your nines are not my nines

#118

Earlier quoted context omitted.

You might want to have a look at some Google Cloud SLAs[1]. They are generally calculated based on actual performance on a particular customer's RPC traffic, down to individual RPCs (at least on Google Cloud Storage, where I work). Read through the agreements to the definition of "Error Rate", and I think you'll find the terms you're looking for. You're welcome! :) The root post does raise an important issue, though…

How are Google Cloud SLAs valuable when the service regularly has multi-hour (IIRC, > 5 hours!) (sometimes global) outages? I pose that Google is not presently in a good position to highlight as a role model or case-study for demonstration of effective cloud provider SLAs.

You report to them your outage (https://support.google.com/cloud/contact/cloud_platform_sla) and then get monetary credits. e.g. for an arbitrarily-clicked service [1], it's 10% off for a month when they don't hit three 9s, 25% if they don't hit two 9s, and 50% off if they're below 95%. Which honestly isn't a very high uptime requirement, but there is a very clear process for getting payouts when they miss it.

[1] https://cloud.google.com/filestore/sla

Re: Your nines are not my nines

#119
post #106

Earlier quoted context omitted.

How are Google Cloud SLAs valuable when the service regularly has multi-hour (IIRC, > 5 hours!) (sometimes global) outages? I pose that Google is not presently in a good position to highlight as a role model or case-study for demonstration of effective cloud provider SLAs.

>How are Google Cloud SLAs valuable when the service regularly has multi-hour (IIRC, > 5 hours!) (sometimes global) outages? You can sue them? (if that's not forbidden by the contract).

See my sibling comment - they have a process for talking to them to get discounts for months when they don't meet the SLA. If they don't give you said discount, you can sue them for breach of contract, but if they pay out you can't sue them for damages.

Re: Your nines are not my nines

#120
post #15

Earlier quoted context omitted.

To get a proper SLA you need to pay for SLA. What SLA google, azure, aws have is useless, some service credit proportional to the outage duration. Totally nothing to cover lost profits or direct damage. For the last gcloud outage, i think you have to talk to people and APPLY for a credit, obviously very few did that https://news.ycombinator.com/item?id=20078296

Working with enterprises that actually have individual SLA's with one of the clouds you mentioned, this is not true. You can handle out your own SLA's with certain providers, and not just get "credits." These enterprises have mature enough monitoring solutions to be able to prove to the provider that they didn't hold up on their end. Besides that, every half-way capable solution architect wouldn't move "system critic…

google cloud compute clearly states that their free sla benefits come in form of credits https://cloud.google.com/compute/sla

amazon ec2 clearly states their sla gives you credits https://aws.amazon.com/compute/sla/

azure compute clearly states they give you credits https://azure.microsoft.com/en-us/support/legal/sla/virtual-...

wanna better sla - pay up, like i said in the beginning. as the cost of sla is proportional to payout that works like an insurance, not like coercive measure to increase reliability.

Post reply on HN