Earlier quoted context omitted.
You may be confused. "One nine" would mean 90% availability.
my bad. 5 nines v.s. 3 nines.
a 40% drop in nines. 40% less reliable!
Your nines are not my nines
91–100 of 135 posts
Re: Your nines are not my nines
#92Million times this. Its shocking how "elevated rate of errors for specific endpoint" in your cloud provider status page is actually amplified to be a soft-outage of your product when your writes to disk never return, your databases returning inconsistent data or your orchestration taking some drastic measures for the failing health check. When you have a lot of components in your cloud mix, failure of one stage(netwo…
Although I agree with your assessment that the total uptime of the system is the multiplication of the individual systems, that doesn’t appear to be the point of the article.
> The problem is that they weren't monitoring from the customer's perspective. Had they done that, it would have been clear that oodles of requests from some subset of customers were failing. They would have also realized that certain customers had all of their requests failing.
This is saying that if you are small, all your failing requests are within the 0.001% that the provider is allowed to fail.
I suppose this depends on what how 99.999 uptime is defined in the SLA.
Re: Your nines are not my nines
#93Million times this. Its shocking how "elevated rate of errors for specific endpoint" in your cloud provider status page is actually amplified to be a soft-outage of your product when your writes to disk never return, your databases returning inconsistent data or your orchestration taking some drastic measures for the failing health check. When you have a lot of components in your cloud mix, failure of one stage(netwo…
The fact that many status pages are updated manually and any incident disclosure need to get approval from management(aws?) does not add to the status page trust. Uptime and error metrics are technical and should be kept away from managers.
> The fact that many status pages are updated manually and any incident disclosure need to get approval from management(aws?) does not add to the status page trust.
Automated status page updates can also reduce trust, since then the status page is itself exposed to more kinds of system failures.
Re: Your nines are not my nines
#94Million times this. Its shocking how "elevated rate of errors for specific endpoint" in your cloud provider status page is actually amplified to be a soft-outage of your product when your writes to disk never return, your databases returning inconsistent data or your orchestration taking some drastic measures for the failing health check. When you have a lot of components in your cloud mix, failure of one stage(netwo…
> if 10 of your cloud services each have a reliability of 99.999, all together the reliability is not 99.999. (The answer is 99.99)
It's probably not in practice, since that assumes failures are perfectly independent. If they are perfectly correlated, the answer is still 99.999. For most real cases, it will be between those extremes.
Re: Your nines are not my nines
#95Earlier quoted context omitted.
I worked at a company once where each bug had a really interesting field: root cause I wish I could remember the values you could fill in, they were very intelligently chosen. What I learned: if you didn't know what the root cause was, you probably didn't fix anything.
There is no root cause. https://www.kitchensoap.com/2012/02/10/each-necessary-but-on... https://blog.acolyer.org/2016/02/10/how-complex-systems-fail... http://web.mit.edu/2.75/resources/random/How%20Complex%20Sys...
Another way to look at that is that a root cause can be a set.
Re: Your nines are not my nines
#96Re: Your nines are not my nines
#97I have a hard time taking this article seriously when it’s all “innuendo” and not actually naming any names or providing any verifiable facts at all.
If the author had a specific problem with specific SLA’s, tell us with real details.
And SLA’s aren’t for winning the lottery or providing impossible-to-meet standards. You need to look at what they actually cover, compare with your costs and reliability of running infrastructure in-house, and then pick the right tradeoff for you. I can’t even tell if the author is accusing cloud providers of fraud, of being misleading, if the author just never understood the SLA properly, or what.
Re: Your nines are not my nines
#98Earlier quoted context omitted.
The fact that many status pages are updated manually and any incident disclosure need to get approval from management(aws?) does not add to the status page trust. Uptime and error metrics are technical and should be kept away from managers.
Maybe it's time for a consumer watchdog group to step in and do their own reporting for services like this. Like https://www.isitdownrightnow.com/ but with sharper teeth. I'm not sure how much time I have to participate but I wouldn't mind chipping in a bit on a co-op in this space. But it might be easier to convince Is it Down Right Now to grow some fangs, or socialize the idea that it does (perception counts for a…
Or it's time to go back to one's own infrastructure and take one's own destiny into one's own hands, along with the responsibility.
Re: Your nines are not my nines
#99This can be complicated, though. I work for a large CDN, and we have systems that monitor our customer experience. Almost every issue those systems discover, however, end up being issues with a customer's origin or configuration. We ended up having to change our procedures on how we responded to issues we discovered because all of our support time was spent checking these issues, and realizing they were outside our c…
I work for a company in a similar situation, and I agree.
The author wants us to look at things from the customer's perspective. The thing is, we (and presumably all major cloud providers) do. Every feature released, every API call, has a canary associated with it that does nothing but pretend to be a customer using that feature. There are definitely cases that slip through the cracks that shouldn't have (we forgot to properly test for a certain condition or combination use case etc) but the vast majority of the time a customer experiences an outage it's because of something the customer did.
That's not to excuse the 5 9 guarantee that inspires fake confidence. But we're always upfront with customers that there's a shared responsibility for availability: it's our responsible to make sure what customers pay for works, but it's also a customer's responsibility that there's enough redundancy in their architecture for their use case.
Re: Your nines are not my nines
#100Earlier quoted context omitted.
There is no root cause. https://www.kitchensoap.com/2012/02/10/each-necessary-but-on... https://blog.acolyer.org/2016/02/10/how-complex-systems-fail... http://web.mit.edu/2.75/resources/random/How%20Complex%20Sys...
Another way to look at that is that a root cause can be a set.
I've had more than one case at work where it would come down to bad requirements. Both systems worked exactly as specified and were bug free (for the issue in scope). They just had a different understanding of reality by design. Root cause here is some mixture of poor understanding of the problem domain by various staff.