Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Your nines are not my nines

rachelbythebay.com

Post reply on HN
Print view View original

61–70 of 135 posts

Re: Your nines are not my nines

#61

Post by jameshart »

hinkley wrote: post #23

We've talked about these classes of problem from time to time, a lot of ideas have been put forward but what's the solution? If I have a multi-tenant system, and no one customer is dominant (always causes problems IMO), my 'biggest customer' might only be 4% of my traffic. There are a million things that can go wrong that make this customer's experiences different from everyone else's, from getting my sharding soluti…

The issue is that percentiles are a very crude tool. This isn’t just a multitenancy problem, it can manifest in any multiuser system. If 0.5% of your traffic comes from New Zealand, a DNS issue affecting your CDN routing that causes all NZ traffic to time out won’t affect your 99th percentile loadtime graph at all. Essentially, percentiles are useless for discovering problems that have a strong effect on a small portion of your traffic.

Scatter plots and histograms are much better at telling you when the distribution of a stat has gone bimodal with a small but consistent group in the outlier group. Percentiles only make sense for telling you, when you already know what shape the distribution curve should be, how flat that curve is right now. They don’t tell you when the shape has changed.

Re: Your nines are not my nines

#62

Post by solipsism »

mentat wrote: post #39
hinkley wrote: post #23

We've talked about these classes of problem from time to time, a lot of ideas have been put forward but what's the solution? If I have a multi-tenant system, and no one customer is dominant (always causes problems IMO), my 'biggest customer' might only be 4% of my traffic. There are a million things that can go wrong that make this customer's experiences different from everyone else's, from getting my sharding soluti…

Having standard shaped telemetry for SDKs (via OpenTelemetry hopefully now) then allowing opt-in aggregation of customer views, either as an observability product to peers or to the providers is what I've been trying to get interest in for the last few years. Having consistent data to show cloud support is also helpful, even if only for one company, especially if you can show you're usually right. A short blog post o…

What role does the word "shaped" have in your opening sentence? Telemetry that is standard-shaped? Standard telemetry that is shaped?

And what does it mean? And am I stupid, or is this not a term everyone knows?

Re: Your nines are not my nines

#63

Post by solipsism »

aluminussoma wrote: post #10

The problem is that the vendor is incentivized to publicly use whatever metric shows the highest availability. Otherwise, the vendor will have to pay back credits. The vendor's nines are never my nines.

No, the vendor is incentived to provide a good experience to the customer, especially vendors that aren't AWS. They know they have everything to lose from customers changing clouds.

Ineptitude, and it being a hard problem, are sufficient to explain the status quo.

Re: Your nines are not my nines

#65

Post by vincentmarle »

altmind wrote: post #12

Million times this. Its shocking how "elevated rate of errors for specific endpoint" in your cloud provider status page is actually amplified to be a soft-outage of your product when your writes to disk never return, your databases returning inconsistent data or your orchestration taking some drastic measures for the failing health check. When you have a lot of components in your cloud mix, failure of one stage(netwo…

> if 10 of your cloud services each have a reliability of 99.999, all together the reliability is not 99.999.

(The answer is 99.99)

Re: Your nines are not my nines

#66

Post by tschellenbach »

I think these blogposts create a false narrative. It should start by acknowledging that in 99% of cases any outsourced systems' stability will be better than anything in-house. Yes there are some vendors that don't do a good job. I heard many people complaining about Layer for instance. But most vendors, AWS, Stripe, Algolia, Stream, can invest more time and effort in stability than you can feasible do for an in-house solution. This is not surprising, if you do something for thousands of customers you can dedicate more effort to Q&A, Docs, maintenance, monitoring, firefighting etc. For every story of someone having vendor issues there are dozens for things going to shit with in-house code.

There have been times in the past when I was annoyed with AWS stability issues. We've all been there. But I also know that AWS is more stable than anything I could feasible build in-house.

Re: Your nines are not my nines

#67

Post by m463 »

finnthehuman wrote: post #53
m463 wrote: post #32

Earlier quoted context omitted.

I worked at a company once where each bug had a really interesting field: root cause I wish I could remember the values you could fill in, they were very intelligently chosen. What I learned: if you didn't know what the root cause was, you probably didn't fix anything.

Was it like an list of predefined values? Where I work they do root cause analysis for everything, but with freeform answers so what you describe might be different from what I'm used to. In general, I'm so used to RCA and layered mitigations (what one of our greybeards calls "belt and suspenders") that I don't know how quality happens without it. I'm a convert to the idea that if you can't fix a problem directly, th…

Unfortunately this was probably 20 years ago. I know it was a list of predefined values in a drop-down, but I'm not sure if there was a other/write-in field.

The gist was that the causes were appropriate and educational. Folks couldn't choose "user is an idiot", instead having to choose "the interface was confusing".

Re: Your nines are not my nines

#68

Post by unreal37 »

013a wrote: post #46
Rafuino wrote: post #19

I've checked and the top cloud players all have uptime SLAs (which according to the blog post don't seem to have the necessary granularity to matter). See https://aws.amazon.com/compute/sla/ , https://cloud.google.com/compute/sla , and https://azure.microsoft.com/en-us/support/legal/sla/summary/ for examples. But are there other SLAs like for in-zone latency, or hardware performance (e.g. IOPS or bandwidth from your…

They all have SLAs, but the "tier" at which a problem happens dramatically affects how the SLA pays out, or if it pays out. For example, within the AWS Compute SLA you linked: > Unavailable is defined as: For Amazon EC2 (other than Single EC2 Instances), Amazon ECS, or Amazon Fargate, when all of your running instances or running tasks, as applicable, deployed in two or more AZs in the same AWS region (or, if there i…

I know it's just an example, but it really IS how you architect your solution. If you have a single EC2 instance without redundancy that's important to something, you're doing it wrong.

Re: Your nines are not my nines

#69

Post by ChuckMcM »

beering wrote: post #55
ChuckMcM wrote: post #35

This rings so true it hurts. At a very large, very blue, company I recall a time trying to explain to an account manager that I wanted to write the SLAs in terms of my footprint, which is to say, given the resources you have allocated to my account, lets set some SLAs like "latency from any node to any node", "latency from any node to the primary internet", "latency from any node to the secondary internet", "availabi…

Would you be willing to pay more to have the SLAs behave like that, and if so, how much more? Genuinely curious, maybe there's a market for "cloud but better SLAs".

Good question. The point of the article is that the value of cloud SLAs are inversely proportional to the size of the cloud.

Think of it this way, consider the definition of an "availability" SLA as 'the mean availability of all hosts in our cloud'. If its reported at "five 9's" or 99.999% that means that a cloud of 100,000 machines could have one machine down for days at a time and never cause their SLA to slip. Big providers average over multiple hundreds of thousands of machines, your stuff could be down all the time and yet 'everything' is "meeting all the SLAs".

You see this outside of data centers in other overly generalized metrics. Unemployment is only 3.7%[1]! Yay right? Tell that to the people of Magoffin County Kentucky where unemployment is 12.3%[2]

So would I pay more? I don't know. If none of the service providers would offer SLAs based on my footprint, it would not be a choice. If one does, then it becomes the preferred choice even if it is more expensive. At which point do all of them to remain competitive? Another good question. Could be a good differentiator for the #3 cloud provider Google. I know they have the technology to do it if they chose to.

[1] https://www.bls.gov/news.release/pdf/empsit.pdf

[2] https://www.lanereport.com/112723/2019/04/state-releases-cou...

Re: Your nines are not my nines

#70

Post by tootie »

user5994461 wrote: post #31
tootie wrote: post #14

I've been on the receiving end of this from the POV of a fortune 50 company. Companies that are not gnats on anyone's window. Treatment is the same. These big guys just all suck at professional service. All their money goes into sales and product engineering. Lock-in breeds retention. The biggest clients are the ones that tend to be the most locked in too.

Having worked in F50 too. When there is a choice to make between public cloud with 99.xxx% SLA or the internal cloud with 90% uptime and 6 months SLA to get a server, the right choice is always cloud.

Yeah, no kidding. Hybrid is great if you can swing it.
Post reply on HN