Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Your nines are not my nines

rachelbythebay.com

Post reply on HN
Print view View original

41–50 of 135 posts

Re: Your nines are not my nines

#41

Post by mentat »

ex3xu wrote: post #8

Isn't this why service-level agreements exist? If the nines of uptime are that important to your business and you don't want to be a gnat on a windshield, you've got to give the vendor some financial incentive to pay attention to you, right? Or is Rachel talking about a situation where you have an SLA in place, but you can't even prove downtime to the vendor because their monitoring software is inadequate?

The cost to your business will always be greater than the maximum refund on the SLA, even if it's a full refund. That's why you're using the provider in the first place, since you can make more money than they're charging.

Re: Your nines are not my nines

#42

Post by DangitBobby »

user5994461 wrote: post #31
tootie wrote: post #14

I've been on the receiving end of this from the POV of a fortune 50 company. Companies that are not gnats on anyone's window. Treatment is the same. These big guys just all suck at professional service. All their money goes into sales and product engineering. Lock-in breeds retention. The biggest clients are the ones that tend to be the most locked in too.

Having worked in F50 too. When there is a choice to make between public cloud with 99.xxx% SLA or the internal cloud with 90% uptime and 6 months SLA to get a server, the right choice is always cloud.

>public cloud

>internal cloud

>the right choice is always cloud

I can't tell if you missed a word or are making a joke...

Re: Your nines are not my nines

#43

Post by ken »

altmind wrote: post #12

Million times this. Its shocking how "elevated rate of errors for specific endpoint" in your cloud provider status page is actually amplified to be a soft-outage of your product when your writes to disk never return, your databases returning inconsistent data or your orchestration taking some drastic measures for the failing health check. When you have a lot of components in your cloud mix, failure of one stage(netwo…

> if 10 of your cloud services each have a reliability of 99.999, all together the reliability is not 99.999.

It's like an episode of Dirk Niblick: https://www.youtube.com/watch?v=bCoGMYV3UPk

Re: Your nines are not my nines

#44

Post by cortesoft »

This can be complicated, though. I work for a large CDN, and we have systems that monitor our customer experience. Almost every issue those systems discover, however, end up being issues with a customer's origin or configuration. We ended up having to change our procedures on how we responded to issues we discovered because all of our support time was spent checking these issues, and realizing they were outside our control.

There is always two sides to these sorts of things.

Re: Your nines are not my nines

#45

Post by rocqua »

mentat wrote: post #41
ex3xu wrote: post #8

Isn't this why service-level agreements exist? If the nines of uptime are that important to your business and you don't want to be a gnat on a windshield, you've got to give the vendor some financial incentive to pay attention to you, right? Or is Rachel talking about a situation where you have an SLA in place, but you can't even prove downtime to the vendor because their monitoring software is inadequate?

The cost to your business will always be greater than the maximum refund on the SLA, even if it's a full refund. That's why you're using the provider in the first place, since you can make more money than they're charging.

An SLA could technically go beyond refunding payments into payout out fines.

Re: Your nines are not my nines

#46

Post by 013a »

Rafuino wrote: post #19

I've checked and the top cloud players all have uptime SLAs (which according to the blog post don't seem to have the necessary granularity to matter). See https://aws.amazon.com/compute/sla/ , https://cloud.google.com/compute/sla , and https://azure.microsoft.com/en-us/support/legal/sla/summary/ for examples. But are there other SLAs like for in-zone latency, or hardware performance (e.g. IOPS or bandwidth from your…

They all have SLAs, but the "tier" at which a problem happens dramatically affects how the SLA pays out, or if it pays out.

For example, within the AWS Compute SLA you linked:

> Unavailable is defined as: For Amazon EC2 (other than Single EC2 Instances), Amazon ECS, or Amazon Fargate, when all of your running instances or running tasks, as applicable, deployed in two or more AZs in the same AWS region (or, if there is only one AZ in the AWS region, that AZ and an AZ in another AWS region) concurrently have no external connectivity.

Get this: Single EC2 instances have an SLA of 90%. Seriously. Its in that article.

In other words; AZ outages rarely see pay out, because you "didn't architect your cloud correctly". And we've been told some nicer variation of this when asking for a reimbursement a few years back. You do have to ask, you know. They could literally automate this process, but they don't. Whatever.

Let's also be clear about the language here: There's no "pay out" at all. What happens is, you get the amazing privilege of not being forced to pay them for a product that didn't work.

That ties directly back to the article; they pay out based on their architecture and SLAs, which are not your architecture and SLAs, unless you perfectly match your architecture to their architecture, which will have gaps, and then you're bought in so hard that you could never leave if you needed to.

Re: Your nines are not my nines

#47

Post by rconti »

user5994461 wrote: post #31
tootie wrote: post #14

I've been on the receiving end of this from the POV of a fortune 50 company. Companies that are not gnats on anyone's window. Treatment is the same. These big guys just all suck at professional service. All their money goes into sales and product engineering. Lock-in breeds retention. The biggest clients are the ones that tend to be the most locked in too.

Having worked in F50 too. When there is a choice to make between public cloud with 99.xxx% SLA or the internal cloud with 90% uptime and 6 months SLA to get a server, the right choice is always cloud.

The SLA is just a metric used to negotiate discounts/credits next time around.

When I was in ops, thinking the world was going to collapse and we were all going to get fired if a service went down, I didn't really get it. It's just a negotiating tactic. It's a cost of doing business.

Re: Your nines are not my nines

#48

Post by gowld »

mentat wrote: post #41
ex3xu wrote: post #8

Isn't this why service-level agreements exist? If the nines of uptime are that important to your business and you don't want to be a gnat on a windshield, you've got to give the vendor some financial incentive to pay attention to you, right? Or is Rachel talking about a situation where you have an SLA in place, but you can't even prove downtime to the vendor because their monitoring software is inadequate?

The cost to your business will always be greater than the maximum refund on the SLA, even if it's a full refund. That's why you're using the provider in the first place, since you can make more money than they're charging.

That's not true. If you get a month refund for a day down, you can still come out ahead. Even so, it's not reasonable to compare to perfection, you should compare to other options.

Re: Your nines are not my nines

#49

Post by Kenji »

cortesoft wrote: post #44

This can be complicated, though. I work for a large CDN, and we have systems that monitor our customer experience. Almost every issue those systems discover, however, end up being issues with a customer's origin or configuration. We ended up having to change our procedures on how we responded to issues we discovered because all of our support time was spent checking these issues, and realizing they were outside our c…

The solution to this is obviously not to change your procedures and your service, but to change your product to make misconfigurations harder to apply and to provide the user with detailed manuals, instructions, warnings, confirmation buttons, etc.

Re: Your nines are not my nines

#50

Post by deaddrop »

samstokes wrote: post #36
hinkley wrote: post #23

We've talked about these classes of problem from time to time, a lot of ideas have been put forward but what's the solution? If I have a multi-tenant system, and no one customer is dominant (always causes problems IMO), my 'biggest customer' might only be 4% of my traffic. There are a million things that can go wrong that make this customer's experiences different from everyone else's, from getting my sharding soluti…

That's the "monitor from the customer's point of view" approach the OP alludes to. If you use tools like Honeycomb [1] that can easily and routinely answer questions like "show me the 95th percentile latencies for each of the 10 customers experiencing the worst latencies ", then situations like you're describing are a lot easier to discover. [1] https://honeycomb.io . Disclaimer: I used to work for them.

>That's the "monitor from the customer's point of view" approach the OP alludes to.

...but now you're in a recursive problem: Who watches the watcher? If the watcher goes down, your insights are gone. Do you devote your entire engineering staff to monitoring, then?

A two-pronged approach would be better: Customer Touch-Point monitoring built into your product and external monitoring should your CTP monitoring go down. If your external monitoring goes down, you still have the CTP, so not all visibility is lost.

Post reply on HN