Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Your nines are not my nines

rachelbythebay.com

Post reply on HN
Print view View original

31–40 of 135 posts

Re: Your nines are not my nines

#31

Post by user5994461 »

tootie wrote: post #14

I've been on the receiving end of this from the POV of a fortune 50 company. Companies that are not gnats on anyone's window. Treatment is the same. These big guys just all suck at professional service. All their money goes into sales and product engineering. Lock-in breeds retention. The biggest clients are the ones that tend to be the most locked in too.

Having worked in F50 too.

When there is a choice to make between public cloud with 99.xxx% SLA or the internal cloud with 90% uptime and 6 months SLA to get a server, the right choice is always cloud.

Re: Your nines are not my nines

#32

Post by m463 »

altmind wrote: post #12

Million times this. Its shocking how "elevated rate of errors for specific endpoint" in your cloud provider status page is actually amplified to be a soft-outage of your product when your writes to disk never return, your databases returning inconsistent data or your orchestration taking some drastic measures for the failing health check. When you have a lot of components in your cloud mix, failure of one stage(netwo…

I worked at a company once where each bug had a really interesting field: root cause

I wish I could remember the values you could fill in, they were very intelligently chosen.

What I learned: if you didn't know what the root cause was, you probably didn't fix anything.

Re: Your nines are not my nines

#34

Post by [deleted] »

perlgeek wrote: post #9
ex3xu wrote: post #8

Isn't this why service-level agreements exist? If the nines of uptime are that important to your business and you don't want to be a gnat on a windshield, you've got to give the vendor some financial incentive to pay attention to you, right? Or is Rachel talking about a situation where you have an SLA in place, but you can't even prove downtime to the vendor because their monitoring software is inadequate?

I think this is about the granularity of SLAs and monitoring. If a provider promises that, overall, 99.5% of all requests will succeed, but the 0.5% errors are all concentrated on some few customers / regions / AZs, customers can have a very bad day. So this is about promising each customer that 99.5% of all their requests will succeed, and monitor in a way that makes sure you can keep that promise.

[deleted]

Re: Your nines are not my nines

#35

Post by ChuckMcM »

This rings so true it hurts. At a very large, very blue, company I recall a time trying to explain to an account manager that I wanted to write the SLAs in terms of my footprint, which is to say, given the resources you have allocated to my account, lets set some SLAs like "latency from any node to any node", "latency from any node to the primary internet", "latency from any node to the secondary internet", "availability of primary internet", "secondary internet" and "blended availability of both."

I had a bunch of these things, all of which were things that were tracked, measured, and monitored, in an existing setup.

Their response was, "We really don't have any way to provide the data for your SLAs, much less actually sign up to enforce them." I suggested that they were not serious about being in the 'cloud' business then. They seemed miffed.

Re: Your nines are not my nines

#36

Post by samstokes »

hinkley wrote: post #23

We've talked about these classes of problem from time to time, a lot of ideas have been put forward but what's the solution? If I have a multi-tenant system, and no one customer is dominant (always causes problems IMO), my 'biggest customer' might only be 4% of my traffic. There are a million things that can go wrong that make this customer's experiences different from everyone else's, from getting my sharding soluti…

That's the "monitor from the customer's point of view" approach the OP alludes to. If you use tools like Honeycomb [1] that can easily and routinely answer questions like "show me the 95th percentile latencies for each of the 10 customers experiencing the worst latencies", then situations like you're describing are a lot easier to discover.

[1] https://honeycomb.io. Disclaimer: I used to work for them.

Re: Your nines are not my nines

#39

Post by mentat »

hinkley wrote: post #23

We've talked about these classes of problem from time to time, a lot of ideas have been put forward but what's the solution? If I have a multi-tenant system, and no one customer is dominant (always causes problems IMO), my 'biggest customer' might only be 4% of my traffic. There are a million things that can go wrong that make this customer's experiences different from everyone else's, from getting my sharding soluti…

Having standard shaped telemetry for SDKs (via OpenTelemetry hopefully now) then allowing opt-in aggregation of customer views, either as an observability product to peers or to the providers is what I've been trying to get interest in for the last few years. Having consistent data to show cloud support is also helpful, even if only for one company, especially if you can show you're usually right. A short blog post on this I wrote a bit ago: https://lightstep.com/blog/tough-conversations-with-cloud-pr...
Post reply on HN