Is this title a consequence of some automatic HN system to try to reduce "fluff" in titles? It's kind of nonsensical in this instance. I saw another title earlier today missing a leading "How" that also didn't make much sense.
[deleted]
Your nines are not my nines
11–20 of 135 posts
Re: Your nines are not my nines
#12Million times this.
Its shocking how "elevated rate of errors for specific endpoint" in your cloud provider status page is actually amplified to be a soft-outage of your product when your writes to disk never return, your databases returning inconsistent data or your orchestration taking some drastic measures for the failing health check.
When you have a lot of components in your cloud mix, failure of one stage(network->balancing->quering->rendering->persistence) bring everything down.
if 10 of your cloud services each have a reliability of 99.999, all together the reliability is not 99.999.
cloud providers can claim mountain-high availablity whereas users will never get their apps running with advertised reliability for now there is multiple subcomponents that can fail.
Re: Your nines are not my nines
#13Million times this. Its shocking how "elevated rate of errors for specific endpoint" in your cloud provider status page is actually amplified to be a soft-outage of your product when your writes to disk never return, your databases returning inconsistent data or your orchestration taking some drastic measures for the failing health check. When you have a lot of components in your cloud mix, failure of one stage(netwo…
The fact that many status pages are updated manually and any incident disclosure need to get approval from management(aws?) does not add to the status page trust.
Uptime and error metrics are technical and should be kept away from managers.
Re: Your nines are not my nines
#14I've been on the receiving end of this from the POV of a fortune 50 company. Companies that are not gnats on anyone's window. Treatment is the same. These big guys just all suck at professional service. All their money goes into sales and product engineering. Lock-in breeds retention. The biggest clients are the ones that tend to be the most locked in too.
Re: Your nines are not my nines
#15Isn't this why service-level agreements exist? If the nines of uptime are that important to your business and you don't want to be a gnat on a windshield, you've got to give the vendor some financial incentive to pay attention to you, right? Or is Rachel talking about a situation where you have an SLA in place, but you can't even prove downtime to the vendor because their monitoring software is inadequate?
To get a proper SLA you need to pay for SLA.
What SLA google, azure, aws have is useless, some service credit proportional to the outage duration. Totally nothing to cover lost profits or direct damage.
For the last gcloud outage, i think you have to talk to people and APPLY for a credit, obviously very few did that https://news.ycombinator.com/item?id=20078296
Re: Your nines are not my nines
#16The problem is that the vendor is incentivized to publicly use whatever metric shows the highest availability. Otherwise, the vendor will have to pay back credits. The vendor's nines are never my nines.
The only way this gets solved is through cloud consumers providing streams of telemetry (sanitized of any data of value besides success/failure metrics of the underlying cloud primitives) to a central reporting uptime stats broker (Speedtest.net meets DataDog meets the Internet Weather Map). The incentives to fudge or exaggerate you uptime claims as a vendor through sales and marketing is too high; let the data speak for itself.
Do you trust AWS' status page? Or are you coming to Hacker News to ask why your network latency between instances has skyrocketed unexpectedly?
Re: Your nines are not my nines
#17This doesn’t seem to be true (at least for all vendors). AFAIK, Google Cloud has per-customer SLAs, though you might need to have enough traffic for statistical significance in some products.
Re: Your nines are not my nines
#18This is so incredibly true. My company has internal services used between teams. For some reason my app can always tell when another app is down and they never can.
Re: Your nines are not my nines
#19I've checked and the top cloud players all have uptime SLAs (which according to the blog post don't seem to have the necessary granularity to matter). See https://aws.amazon.com/compute/sla/, https://cloud.google.com/compute/sla, and https://azure.microsoft.com/en-us/support/legal/sla/summary/ for examples.
But are there other SLAs like for in-zone latency, or hardware performance (e.g. IOPS or bandwidth from your local or remote storage)? Are these kinds of SLAs part of larger private agreements (like, Netflix, a huge AWS customer), or is uptime the only SLA offered? Haven't been able to find any info on this in my searches...
Re: Your nines are not my nines
#20Is this title a consequence of some automatic HN system to try to reduce "fluff" in titles? It's kind of nonsensical in this instance. I saw another title earlier today missing a leading "How" that also didn't make much sense.
Indeed. Sorry! If you notice such disfigurement in the future feel free to email hn@ycombinator.com and we might get to it quicker.