Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Your nines are not my nines

rachelbythebay.com

Post reply on HN
Print view View original

21–30 of 135 posts

Re: Your nines are not my nines

#21

Post by scottmcf »

altmind wrote: post #13
altmind wrote: post #12

Million times this. Its shocking how "elevated rate of errors for specific endpoint" in your cloud provider status page is actually amplified to be a soft-outage of your product when your writes to disk never return, your databases returning inconsistent data or your orchestration taking some drastic measures for the failing health check. When you have a lot of components in your cloud mix, failure of one stage(netwo…

The fact that many status pages are updated manually and any incident disclosure need to get approval from management(aws?) does not add to the status page trust. Uptime and error metrics are technical and should be kept away from managers.

This was extremely evident in the slow response and poor communication during the recent Salesforce outage.

Re: Your nines are not my nines

#23

Post by hinkley »

We've talked about these classes of problem from time to time, a lot of ideas have been put forward but what's the solution?

If I have a multi-tenant system, and no one customer is dominant (always causes problems IMO), my 'biggest customer' might only be 4% of my traffic. There are a million things that can go wrong that make this customer's experiences different from everyone else's, from getting my sharding solution wrong to small-C n^2 issues (and a whole lot of space between for nlog(n) problems).

If I'm doing 95th percentile calculations that will not show up in my metrics. If I have a larger customer that's 10% of my traffic, almost half of their users could be having issues before my alerts go off.

And then there's explaining to your boss that 5 9's across twenty interacting services is around 99.98%, and that's only if degradation in one service doesn't cause failure in another.

Re: Your nines are not my nines

#24

Post by ownagefool »

ex3xu wrote: post #8

Isn't this why service-level agreements exist? If the nines of uptime are that important to your business and you don't want to be a gnat on a windshield, you've got to give the vendor some financial incentive to pay attention to you, right? Or is Rachel talking about a situation where you have an SLA in place, but you can't even prove downtime to the vendor because their monitoring software is inadequate?

It's a bit of both. But also, you need to remember, legal may pass blame, but they won't keep your app up. Your app being down is likely bad, even if you get paid for it.

Re: Your nines are not my nines

#25

Post by hinkley »

altmind wrote: post #12

Million times this. Its shocking how "elevated rate of errors for specific endpoint" in your cloud provider status page is actually amplified to be a soft-outage of your product when your writes to disk never return, your databases returning inconsistent data or your orchestration taking some drastic measures for the failing health check. When you have a lot of components in your cloud mix, failure of one stage(netwo…

Saucelabs is very bad this way. We have tunnels flake out once in a while (I'm still convinced there's a concurrency bug in their tunnel implementation, based on missing events I've seen in test logs), but sometimes Sauce is just having issues.

When I'm seeing 100% failure rate, there's often nothing on their status page. Or there's some bullshit metric like VM acquisition times are double normal for, say, some Windows VM. But I'm not seeing 8% failure rate. I'm not seeing an extra 30 seconds. I'm seeing 100% failure rate, with long timeouts, and retries.

Re: Your nines are not my nines

#26

Post by hinkley »

altmind wrote: post #13
altmind wrote: post #12

Million times this. Its shocking how "elevated rate of errors for specific endpoint" in your cloud provider status page is actually amplified to be a soft-outage of your product when your writes to disk never return, your databases returning inconsistent data or your orchestration taking some drastic measures for the failing health check. When you have a lot of components in your cloud mix, failure of one stage(netwo…

The fact that many status pages are updated manually and any incident disclosure need to get approval from management(aws?) does not add to the status page trust. Uptime and error metrics are technical and should be kept away from managers.

Maybe it's time for a consumer watchdog group to step in and do their own reporting for services like this. Like https://www.isitdownrightnow.com/ but with sharper teeth.

I'm not sure how much time I have to participate but I wouldn't mind chipping in a bit on a co-op in this space.

But it might be easier to convince Is it Down Right Now to grow some fangs, or socialize the idea that it does (perception counts for a lot).

Re: Your nines are not my nines

#28

Post by hinkley »

aluminussoma wrote: post #10

The problem is that the vendor is incentivized to publicly use whatever metric shows the highest availability. Otherwise, the vendor will have to pay back credits. The vendor's nines are never my nines.

"It is difficult to get a man to understand something, when his salary depends on his not understanding it."

As an aside, once in a while I imagine what kind of field day Upton Sinclair would have with this trip around of the pendulum swing toward dystopia.

Re: Your nines are not my nines

#29

Post by hinkley »

toomuchtodo wrote: post #16
aluminussoma wrote: post #10

The problem is that the vendor is incentivized to publicly use whatever metric shows the highest availability. Otherwise, the vendor will have to pay back credits. The vendor's nines are never my nines.

The only way this gets solved is through cloud consumers providing streams of telemetry (sanitized of any data of value besides success/failure metrics of the underlying cloud primitives) to a central reporting uptime stats broker (Speedtest.net meets DataDog meets the Internet Weather Map). The incentives to fudge or exaggerate you uptime claims as a vendor through sales and marketing is too high; let the data speak…

I wonder if this is a sort of thing you could interest EFF or another organization to put funding behind.
Post reply on HN