i'm not sure how this person has their system architected but they should look closely at the 9's the company is talking about. Is it 5 9's across all regions? Within a single region? What about for the specific service? It really all depends, the post is an over simplification or they haven't architected their system appropriately to cover actually get 5 nines on the host cloud.
Your nines are not my nines
71–80 of 135 posts
Re: Your nines are not my nines
#72Million times this. Its shocking how "elevated rate of errors for specific endpoint" in your cloud provider status page is actually amplified to be a soft-outage of your product when your writes to disk never return, your databases returning inconsistent data or your orchestration taking some drastic measures for the failing health check. When you have a lot of components in your cloud mix, failure of one stage(netwo…
I worked at a company once where each bug had a really interesting field: root cause I wish I could remember the values you could fill in, they were very intelligently chosen. What I learned: if you didn't know what the root cause was, you probably didn't fix anything.
Re: Your nines are not my nines
#73Earlier quoted context omitted.
Having worked in F50 too. When there is a choice to make between public cloud with 99.xxx% SLA or the internal cloud with 90% uptime and 6 months SLA to get a server, the right choice is always cloud.
>public cloud >internal cloud >the right choice is always cloud I can't tell if you missed a word or are making a joke...
"internal cloud" is not the cloud. It's just a bottle full of mist.
Re: Your nines are not my nines
#74I've checked and the top cloud players all have uptime SLAs (which according to the blog post don't seem to have the necessary granularity to matter). See https://aws.amazon.com/compute/sla/ , https://cloud.google.com/compute/sla , and https://azure.microsoft.com/en-us/support/legal/sla/summary/ for examples. But are there other SLAs like for in-zone latency, or hardware performance (e.g. IOPS or bandwidth from your…
They all have SLAs, but the "tier" at which a problem happens dramatically affects how the SLA pays out, or if it pays out. For example, within the AWS Compute SLA you linked: > Unavailable is defined as: For Amazon EC2 (other than Single EC2 Instances), Amazon ECS, or Amazon Fargate, when all of your running instances or running tasks, as applicable, deployed in two or more AZs in the same AWS region (or, if there i…
Wow, I missed the 90% uptime for a single instance. That's pretty terrible.
Re: Your nines are not my nines
#75I've been on the receiving end of this from the POV of a fortune 50 company. Companies that are not gnats on anyone's window. Treatment is the same. These big guys just all suck at professional service. All their money goes into sales and product engineering. Lock-in breeds retention. The biggest clients are the ones that tend to be the most locked in too.
Having worked in F50 too. When there is a choice to make between public cloud with 99.xxx% SLA or the internal cloud with 90% uptime and 6 months SLA to get a server, the right choice is always cloud.
So naturally internal cloud always gets picked.
Re: Your nines are not my nines
#76We've talked about these classes of problem from time to time, a lot of ideas have been put forward but what's the solution? If I have a multi-tenant system, and no one customer is dominant (always causes problems IMO), my 'biggest customer' might only be 4% of my traffic. There are a million things that can go wrong that make this customer's experiences different from everyone else's, from getting my sharding soluti…
The issue is that percentiles are a very crude tool. This isn’t just a multitenancy problem, it can manifest in any multiuser system. If 0.5% of your traffic comes from New Zealand, a DNS issue affecting your CDN routing that causes all NZ traffic to time out won’t affect your 99th percentile loadtime graph at all. Essentially, percentiles are useless for discovering problems that have a strong effect on a small port…
What scatter plot and/or histogram do you have in mind to dig up the small but consistent group in the outlier group?
Re: Your nines are not my nines
#77They are not the only ones who are not monitoring from customer perspective. It is actually hard, as it involves much more than watching some metric.
I proposed a user experience monitoring system for my ex employer, which was based on a very simple principle: problem is a difference between what customer wants and should be able to do and what customer is able to do right now.
It kind of funny, but some major eng companies (maybe even all) don’t know what is user experience...
Re: Your nines are not my nines
#78Million times this. Its shocking how "elevated rate of errors for specific endpoint" in your cloud provider status page is actually amplified to be a soft-outage of your product when your writes to disk never return, your databases returning inconsistent data or your orchestration taking some drastic measures for the failing health check. When you have a lot of components in your cloud mix, failure of one stage(netwo…
I worked at a company once where each bug had a really interesting field: root cause I wish I could remember the values you could fill in, they were very intelligently chosen. What I learned: if you didn't know what the root cause was, you probably didn't fix anything.
1000% this.
I've been in many orgs where root cause was either completely missing or completely missed the point. Recently I quit a company that thought they were doing RCA. The fact they sent out an email that there was an outage and then N number of hours or days later another email to say it was "fixed" and that they scaled or thought they found the problem and don't worry about it anymore. Literally we had weeks where the exact same outage occurred multiple times. And with every outage the exact same response.
So... I asked simple questions of leadership as to why RCA process was not implemented. Why RCA did not require a standardized template to be filled out as part of a production outage. Why a "5 Whys" approach wasn't being considered to truly expose the actual root cause. Why there wasn't any accountability.
At the end of the day failure cause is not root cause and many struggle conflating the two. Honestly when an organization doesn't hold true RCA as a critical part of engineering process I, personally, feel as though that organization will innevitably hit a glass ceiling. Among other problem areas the disconnect with RCA, for me, was why I couldn't stay at that company anymore. It was embarrassing watching from the inside as the same mistakes were made over and over with nobody the wiser.
Re: Your nines are not my nines
#79You can get closer to their nines by using less of their stuff - if you use every service aws offers to power a single app, you'll have markedly less 9's than if you only use a few...
The foundational services (VMs, dns, s3, etc) I've found to be more reliable than others (ebs).
Re: Your nines are not my nines
#80Earlier quoted context omitted.
They all have SLAs, but the "tier" at which a problem happens dramatically affects how the SLA pays out, or if it pays out. For example, within the AWS Compute SLA you linked: > Unavailable is defined as: For Amazon EC2 (other than Single EC2 Instances), Amazon ECS, or Amazon Fargate, when all of your running instances or running tasks, as applicable, deployed in two or more AZs in the same AWS region (or, if there i…
Wow, I missed the 90% uptime for a single instance. That's pretty terrible.
It's not 90% uptime, it's 90% SLA. As in the threshold for "down for more than that, we pay out".