Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Your nines are not my nines

rachelbythebay.com

Post reply on HN
Print view View original

81–90 of 135 posts

Re: Your nines are not my nines

#81

Post by astine »

devdas wrote: post #72
m463 wrote: post #32

Earlier quoted context omitted.

I worked at a company once where each bug had a really interesting field: root cause I wish I could remember the values you could fill in, they were very intelligently chosen. What I learned: if you didn't know what the root cause was, you probably didn't fix anything.

There is no root cause. https://www.kitchensoap.com/2012/02/10/each-necessary-but-on... https://blog.acolyer.org/2016/02/10/how-complex-systems-fail... http://web.mit.edu/2.75/resources/random/How%20Complex%20Sys...

The OP is talking about bugs not complex system failures. These are not the same thing. It might take multiple bugs to cause a complex system to fail, because complex systems often have enormous amounts of complexity built in, but a single bug is often a fairly unidimensional thing. You can identify the root cause of a bug.

Re: Your nines are not my nines

#82

Post by stefco_ »

Cloud solutions have plenty of issues, but I'm quite surprised there aren't more replies talking about how many impossible problems have been made tractable and reliable thanks to the cloud (or to be more precise, made much cheaper to solve reliably). The article makes a great point about accountability, in the sense that no one at a cloud-providing corp is immensely worried about transient failures that only affect small user sets heavily (which sucks if you're affected). But for my scientific computing use cases, getting things working reliably at any sort of scale within budget is impossible without the cloud. Research institutions' computing clusters are just smaller, less reliable, less flexible versions of the cloud (good luck getting sysadmins to do anything useful at all).

One of the collaborations I work in, LIGO, recently gave up on private servers and transitioned to AWS for our Gravitational Candidate Database [1] because the cloud is so much better. I made this change to my own low-latency search framework [2] years ago. If you're not "lucky" enough to (be forced to) use a university/collaboration cluster, you'd have to maintain your own server, which is orders of magnitude less reliable and more expensive/difficult. I understand that not all workflows are the same, but for all of my nontrivial applications, cloud providers save so much time and money that I can do something as bold as making a provider-agnostic architecture with more robust failover. I recognize that more complicated workflows might require e.g. 10 separate AWS services with AWS-specific features causing lock-in, but at that level of complexity, I'm guessing the problem must be virtually impossible with a non-cloud solution anyway. If you really can't figure out another way to deal with resiliency, you might just need to accept that your problem space is really hard and that you're lucky to even be able to run it at all. Again, I think the original article is right about the fact that you have to account for this yourself; the cloud is not magic, and your code still has to understand that it is (like all abstractions) going to leak.

Again, the point about responsiveness in the original article is very well-taken; I'm just surprised more people aren't observing that overall the reliability, cost, and flexibility provided by cloud solutions is utterly transformative in terms of reliability.

[1] http://gracedb.ligo.org

[2] http://multimessenger.science

Re: Your nines are not my nines

#83

Post by hermitdev »

I'm testing changes to a process that uses Azure. We have dozens of on site SQL server dbs, but this one process decided to use Cosmos, because...they didnt have to writet as much code. The developer is gone, but I'm left supporting a process that sucks data down from its source, does a little transform, then shoves it into Cosmos. Then, pulls the data directly back from Cosmos to load it to our internal server dbs. Why?! It's a total facepalm to me. Extra stage, extra step and extra complexity and extra cost for no gain. We don't serve any external pages or services. No reason for this data to be in the cloud. All internal use.

Fucking hate devs that do this, especially the ones that wander on before they have to justify their actions to anyone.

/rant

Re: Your nines are not my nines

#84

Post by johngalt »

It's not just the cloud providers perspective on outages. That is the rosiest interpretation of misleading availability stats. There is an obvious moral hazard involved because most availability tracking is self reported and outage criteria are vague.

This behavior certainly isn't limited to cloud providers. If anything internal operations departments are worse. The only difference is that internal departments can be pressured more effectively.

Re: Your nines are not my nines

#85

Post by _bxg1 »

This plays into something I've been thinking about recently, which is that even when a technology scales indefinitely, maybe technology business doesn't. Maybe, in a world where hosting and CRUD apps and everything feel like solved problems, there's still a place for smaller providers that can interface with their customers directly and tailor themselves to their needs. It's a vaguely comforting thought.

Re: Your nines are not my nines

#86

Post by azernik »

ChuckMcM wrote: post #69
beering wrote: post #55

Earlier quoted context omitted.

Would you be willing to pay more to have the SLAs behave like that, and if so, how much more? Genuinely curious, maybe there's a market for "cloud but better SLAs".

Good question. The point of the article is that the value of cloud SLAs are inversely proportional to the size of the cloud. Think of it this way, consider the definition of an "availability" SLA as 'the mean availability of all hosts in our cloud'. If its reported at "five 9's" or 99.999% that means that a cloud of 100,000 machines could have one machine down for days at a time and never cause their SLA to slip. Big…

In the old phone model, SLAs are per-customer: it doesn't matter what your network as a whole looks like, if any phone line goes below N 9s you're going to be paying out the contract penalties. It's still better for the telco to fail for fewer customers, because then they don't have to pay out as much.

The granularity may be harder to define for cloud services, but it is very much doable; it's all about making sure that the target metrics have zero connection to the global state of the system.

Re: Your nines are not my nines

#87

Post by wrs »

inlined wrote: post #17

This doesn’t seem to be true (at least for all vendors). AFAIK, Google Cloud has per-customer SLAs, though you might need to have enough traffic for statistical significance in some products.

Using GCE as an example [0], it’s per-customer, but all instances in multiple zones have to be unavailable. You could have 99% instance failure and not qualify.

>Loss of external connectivity or persistent disk access for all running Instances, when Instances are placed across two or more Zones in the same Region.

[0] https://cloud.google.com/compute/sla

Re: Your nines are not my nines

#88

Post by rossjudson »

ChuckMcM wrote: post #69
beering wrote: post #55

Earlier quoted context omitted.

Would you be willing to pay more to have the SLAs behave like that, and if so, how much more? Genuinely curious, maybe there's a market for "cloud but better SLAs".

Good question. The point of the article is that the value of cloud SLAs are inversely proportional to the size of the cloud. Think of it this way, consider the definition of an "availability" SLA as 'the mean availability of all hosts in our cloud'. If its reported at "five 9's" or 99.999% that means that a cloud of 100,000 machines could have one machine down for days at a time and never cause their SLA to slip. Big…

You might want to have a look at some Google Cloud SLAs[1]. They are generally calculated based on actual performance on a particular customer's RPC traffic, down to individual RPCs (at least on Google Cloud Storage, where I work). Read through the agreements to the definition of "Error Rate", and I think you'll find the terms you're looking for.

You're welcome! :)

The root post does raise an important issue, though -- just because GCS thinks it's doing great on your RPCs doesn't mean that your system is doing great.

[1] https://cloud.google.com/terms/sla/

Re: Your nines are not my nines

#89

Post by rossjudson »

ropman76 wrote: post #52
cortesoft wrote: post #44

This can be complicated, though. I work for a large CDN, and we have systems that monitor our customer experience. Almost every issue those systems discover, however, end up being issues with a customer's origin or configuration. We ended up having to change our procedures on how we responded to issues we discovered because all of our support time was spent checking these issues, and realizing they were outside our c…

One of the most frustrating things about dealing with situations like this is actually getting ahold of someone with enough experience to say where the issue is to begin with even if it’s out of the provider’s control. I have sent a lot of log files to cloud vendor trying to find why their web hosted application was so slow (6-10 second response times on a CRM app they provided). If someone would have responded with…

I think there is a lot of room for cloud provider innovation in this area. It shouldn't take a human to tell you what's wrong.

Re: Your nines are not my nines

#90

Post by jaytaylor »

rossjudson wrote: post #88
ChuckMcM wrote: post #69

Earlier quoted context omitted.

Good question. The point of the article is that the value of cloud SLAs are inversely proportional to the size of the cloud. Think of it this way, consider the definition of an "availability" SLA as 'the mean availability of all hosts in our cloud'. If its reported at "five 9's" or 99.999% that means that a cloud of 100,000 machines could have one machine down for days at a time and never cause their SLA to slip. Big…

You might want to have a look at some Google Cloud SLAs[1]. They are generally calculated based on actual performance on a particular customer's RPC traffic, down to individual RPCs (at least on Google Cloud Storage, where I work). Read through the agreements to the definition of "Error Rate", and I think you'll find the terms you're looking for. You're welcome! :) The root post does raise an important issue, though…

How are Google Cloud SLAs valuable when the service regularly has multi-hour (IIRC, > 5 hours!) (sometimes global) outages?

I pose that Google is not presently in a good position to highlight as a role model or case-study for demonstration of effective cloud provider SLAs.

Post reply on HN