Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Your nines are not my nines

rachelbythebay.com

Post reply on HN
Print view View original

51–60 of 135 posts

Re: Your nines are not my nines

#51

Post by merb »

user5994461 wrote: post #31
tootie wrote: post #14

I've been on the receiving end of this from the POV of a fortune 50 company. Companies that are not gnats on anyone's window. Treatment is the same. These big guys just all suck at professional service. All their money goes into sales and product engineering. Lock-in breeds retention. The biggest clients are the ones that tend to be the most locked in too.

Having worked in F50 too. When there is a choice to make between public cloud with 99.xxx% SLA or the internal cloud with 90% uptime and 6 months SLA to get a server, the right choice is always cloud.

I'm not sure but managing 1000s of servers/storage whatever manually is quiet a feat.

Re: Your nines are not my nines

#52

Post by ropman76 »

cortesoft wrote: post #44

This can be complicated, though. I work for a large CDN, and we have systems that monitor our customer experience. Almost every issue those systems discover, however, end up being issues with a customer's origin or configuration. We ended up having to change our procedures on how we responded to issues we discovered because all of our support time was spent checking these issues, and realizing they were outside our c…

One of the most frustrating things about dealing with situations like this is actually getting ahold of someone with enough experience to say where the issue is to begin with even if it’s out of the provider’s control.

I have sent a lot of log files to cloud vendor trying to find why their web hosted application was so slow (6-10 second response times on a CRM app they provided). If someone would have responded with an actual answer (your firewall is blocking traffic or try this setup etc) I could have worked with that. Instead we got nothing but stealth ticket closes and “sorry we don’t know why this is slow” responses. This article hit a nerve because you really do dance to someone else’s tune when you go to the “cloud”.

Re: Your nines are not my nines

#53

Post by finnthehuman »

m463 wrote: post #32
altmind wrote: post #12

Million times this. Its shocking how "elevated rate of errors for specific endpoint" in your cloud provider status page is actually amplified to be a soft-outage of your product when your writes to disk never return, your databases returning inconsistent data or your orchestration taking some drastic measures for the failing health check. When you have a lot of components in your cloud mix, failure of one stage(netwo…

I worked at a company once where each bug had a really interesting field: root cause I wish I could remember the values you could fill in, they were very intelligently chosen. What I learned: if you didn't know what the root cause was, you probably didn't fix anything.

Was it like an list of predefined values? Where I work they do root cause analysis for everything, but with freeform answers so what you describe might be different from what I'm used to.

In general, I'm so used to RCA and layered mitigations (what one of our greybeards calls "belt and suspenders") that I don't know how quality happens without it. I'm a convert to the idea that if you can't fix a problem directly, the fix has to isolate or be as close to the problem as possible. Otherwise the bad state just ripples outward as complexity.

Re: Your nines are not my nines

#54

Post by ogn3rd »

ropman76 wrote: post #52
cortesoft wrote: post #44

This can be complicated, though. I work for a large CDN, and we have systems that monitor our customer experience. Almost every issue those systems discover, however, end up being issues with a customer's origin or configuration. We ended up having to change our procedures on how we responded to issues we discovered because all of our support time was spent checking these issues, and realizing they were outside our c…

One of the most frustrating things about dealing with situations like this is actually getting ahold of someone with enough experience to say where the issue is to begin with even if it’s out of the provider’s control. I have sent a lot of log files to cloud vendor trying to find why their web hosted application was so slow (6-10 second response times on a CRM app they provided). If someone would have responded with…

"because you really do dance to someone else’s tune when you go to the “cloud”"

The same can be said of large orgs with a large on-prem footprint.

Re: Your nines are not my nines

#55

Post by beering »

ChuckMcM wrote: post #35

This rings so true it hurts. At a very large, very blue, company I recall a time trying to explain to an account manager that I wanted to write the SLAs in terms of my footprint, which is to say, given the resources you have allocated to my account, lets set some SLAs like "latency from any node to any node", "latency from any node to the primary internet", "latency from any node to the secondary internet", "availabi…

Would you be willing to pay more to have the SLAs behave like that, and if so, how much more? Genuinely curious, maybe there's a market for "cloud but better SLAs".

Re: Your nines are not my nines

#56

Post by finnthehuman »

hinkley wrote: post #23

We've talked about these classes of problem from time to time, a lot of ideas have been put forward but what's the solution? If I have a multi-tenant system, and no one customer is dominant (always causes problems IMO), my 'biggest customer' might only be 4% of my traffic. There are a million things that can go wrong that make this customer's experiences different from everyone else's, from getting my sharding soluti…

>There are a million things that can go wrong that make this customer's experiences different from everyone else

In my experience the things that are easier to write-off as unique to one case, non-representative, or too rare to fix, so they don't have to be thoroughly addressed are warning signs of a robustness issue. Still doesn't mean that they'll get fixed then and there, but they often come around to bite you in the ass later.

Re: Your nines are not my nines

#57

Post by shanemhansen »

I think the fundamental problem, in engineering terms, is that most cloud deployments effectively wire together cloud components serially.

Your LB may have some nines, your individual vms (or set of vms in a region) may have some nines, your data store may have some nines, but if all of them aren't working together it's unlikely your business will be up.

This is inherently customer-dependent and yet it's super predictable (nobody only uses a lb).

Post reply on HN