Earlier quoted context omitted.
I can't remember ever seeing this work out well lol. Happy to be proven wrong one day, though.
You may have heard of a company called Amazon.
It appears I had forgotten about that back-story!
Your nines are not my nines
121–130 of 135 posts
Re: Your nines are not my nines
#122Million times this. Its shocking how "elevated rate of errors for specific endpoint" in your cloud provider status page is actually amplified to be a soft-outage of your product when your writes to disk never return, your databases returning inconsistent data or your orchestration taking some drastic measures for the failing health check. When you have a lot of components in your cloud mix, failure of one stage(netwo…
I worked at a company once where each bug had a really interesting field: root cause I wish I could remember the values you could fill in, they were very intelligently chosen. What I learned: if you didn't know what the root cause was, you probably didn't fix anything.
> if you didn't know what the root cause was, you probably didn't fix anything.
Yes! If you don't know root cause, then you don't know what went wrong. Not only do you not know what to fix, any shotgun debugging is likely to have only fixed one symptom, leaving the actual malfunction in place.
Almost all of the companies I've owned or worked for have recognized this with a simple rule: if you haven't found (and proven) root cause, then the bug cannot be closed as fixed. Any company that doesn't have a variation of this going on is a company whose products you can't trust (and a company I would prefer not to work for).
Re: Your nines are not my nines
#123Earlier quoted context omitted.
I worked at a company once where each bug had a really interesting field: root cause I wish I could remember the values you could fill in, they were very intelligently chosen. What I learned: if you didn't know what the root cause was, you probably didn't fix anything.
Was it like an list of predefined values? Where I work they do root cause analysis for everything, but with freeform answers so what you describe might be different from what I'm used to. In general, I'm so used to RCA and layered mitigations (what one of our greybeards calls "belt and suspenders") that I don't know how quality happens without it. I'm a convert to the idea that if you can't fix a problem directly, th…
I like having a set of broad predefined values (it helps with standardization, which helps with searching in the future).
But a freeform report is also necessary. How else are you going to adequately explain what, where, why, etc., the root cause was?
Re: Your nines are not my nines
#124Earlier quoted context omitted.
Another way to look at that is that a root cause can be a set.
I've had more than one case at work where it would come down to bad requirements. Both systems worked exactly as specified and were bug free (for the issue in scope). They just had a different understanding of reality by design. Root cause here is some mixture of poor understanding of the problem domain by various staff.
> Both systems worked exactly as specified and were bug free
If the system is working as designed, then there really is no bug. A bug is a malfunction, after all.
I've always called broken systems that are working as designed BAD: Broken As Designed.
Re: Your nines are not my nines
#125I think these blogposts create a false narrative. It should start by acknowledging that in 99% of cases any outsourced systems' stability will be better than anything in-house. Yes there are some vendors that don't do a good job. I heard many people complaining about Layer for instance. But most vendors, AWS, Stripe, Algolia, Stream, can invest more time and effort in stability than you can feasible do for an in-hous…
> It should start by acknowledging that in 99% of cases any outsourced systems' stability will be better than anything in-house.
I can't acknowledge this, as it has not been my experience at all.
Re: Your nines are not my nines
#126This can be complicated, though. I work for a large CDN, and we have systems that monitor our customer experience. Almost every issue those systems discover, however, end up being issues with a customer's origin or configuration. We ended up having to change our procedures on how we responded to issues we discovered because all of our support time was spent checking these issues, and realizing they were outside our c…
Where I currently work (all of our customers are enterprises), we encounter this all the time. I'd say about 90% of our serious support tickets are problems with the customer's system or code and are completely out of our control.
However, we will spend a great deal of time resolving their issues regardless. Last week, for example, we had a customer encountering failures with their program using our product. I obtained the source for the customer's application and debugged it for them.
I like that we do this -- it's really nice to solve a customer's problem, and even nicer to be able to tell them it wasn't the fault of our software. It's expensive, of course, but our support contracts are priced to take this into account.
Re: Your nines are not my nines
#127Earlier quoted context omitted.
I've had more than one case at work where it would come down to bad requirements. Both systems worked exactly as specified and were bug free (for the issue in scope). They just had a different understanding of reality by design. Root cause here is some mixture of poor understanding of the problem domain by various staff.
> Both systems worked exactly as specified and were bug free If the system is working as designed, then there really is no bug. A bug is a malfunction, after all. I've always called broken systems that are working as designed BAD: Broken As Designed.
I'd agree. The defect (not sure you can call it a bug?) was in the design/spec not the system.
Re: Your nines are not my nines
#128Earlier quoted context omitted.
They all have SLAs, but the "tier" at which a problem happens dramatically affects how the SLA pays out, or if it pays out. For example, within the AWS Compute SLA you linked: > Unavailable is defined as: For Amazon EC2 (other than Single EC2 Instances), Amazon ECS, or Amazon Fargate, when all of your running instances or running tasks, as applicable, deployed in two or more AZs in the same AWS region (or, if there i…
I know it's just an example, but it really IS how you architect your solution. If you have a single EC2 instance without redundancy that's important to something, you're doing it wrong.
You're not wrong, but it feels like a cheap answer. Why should I pay twice as much just because AWS can't keep an instance up? (And double-cost can be understating it; I've run commercial software where the multi-instance/clustered version is far more expensive than the single-node version. I'd actually like to make it fully multi-AZ, but I'm not gonna get the company to to drop that kind of money on it.)
Re: Your nines are not my nines
#129Earlier quoted context omitted.
Working with enterprises that actually have individual SLA's with one of the clouds you mentioned, this is not true. You can handle out your own SLA's with certain providers, and not just get "credits." These enterprises have mature enough monitoring solutions to be able to prove to the provider that they didn't hold up on their end. Besides that, every half-way capable solution architect wouldn't move "system critic…
google cloud compute clearly states that their free sla benefits come in form of credits https://cloud.google.com/compute/sla amazon ec2 clearly states their sla gives you credits https://aws.amazon.com/compute/sla/ azure compute clearly states they give you credits https://azure.microsoft.com/en-us/support/legal/sla/virtual-... wanna better sla - pay up, like i said in the beginning. as the cost of sla is proportion…
Did you skip there part where I said enterprises get separate SLA's from the ones you and I get?
Re: Your nines are not my nines
#130Earlier quoted context omitted.
1000% this. I've been in many orgs where root cause was either completely missing or completely missed the point. Recently I quit a company that thought they were doing RCA. The fact they sent out an email that there was an outage and then N number of hours or days later another email to say it was "fixed" and that they scaled or thought they found the problem and don't worry about it anymore. Literally we had weeks…
So how did the leadership respond to your questions? Did they agree, and did things change?
They "took it as feedback". The problem was the leadership was convinced they had a real RCA process in place. They didn't.