21–30 of 108 posts
Here's Twitter Back off decider implementation (Java)
https://github.com/twitter/commons/blob/master/src/java/com/...
When last time i looked i was little clueless on this. Now I find its usage.
An automatic 100% credit for 10 days usage, thats pretty good IMO
Well yes, except that that is usually peanuts compared to the lost income from your service being down. Really the only purpose of a SLA penalty is to incentivize the provider to keep the network reliable.
An automatic 100% credit for 10 days usage, thats pretty good IMO
Well yes, except that that is usually peanuts compared to the lost income from your service being down. Really the only purpose of a SLA penalty is to incentivize the provider to keep the network reliable.
But that's just in general.
When negotiating bespoke SLA penalty clauses, it can be very illuminating for both sides to discuss lost profit + lost confidence + additional costs to the customer and suggest that these be factored in to the penalty clause.
My experience: both the customer and supplier tend to take a deep breath to evaluate whether this deal is a good one for either of them and begin to reassess their level of risk.
In a off-the-shelf service like Amazon, you as a customer are welcome to suggest a change of penalty to your Amazon account manager, and unless you're something like the US government, you will probably be directed to other cloud providers or your own internal IT organisation!
tldr: ""The trigger for this event was a network configuration change. We will audit our change process and increase the automation to prevent this mistake from happening in the future." AMZN has gotten a lot of flack over this outage, and rightly so. But I do want to dissuade anyone from thinking anybody else could do much better. I worked there 10 years ago, when they were closer to 200 engineers, and the caliber o…
Their "control plane" network for the EBS clusters span availability zones in a region? If so, this would be the fatal flaw.
I find it surprising that they did not and do not plan to employ any sort of interlocks/padded walls. What I mean is, if the system is exhibiting some very abnormal state (e.g #remirror_event above a fixed threshold or more than x standard deviations above average) then automated repair actions should probably stop and the issue should be escalated to a human.
tl;dr version?
tldr: ""The trigger for this event was a network configuration change. We will audit our change process and increase the automation to prevent this mistake from happening in the future." AMZN has gotten a lot of flack over this outage, and rightly so. But I do want to dissuade anyone from thinking anybody else could do much better. I worked there 10 years ago, when they were closer to 200 engineers, and the caliber o…
Agreed. It has actually given a very nice excuse to developers like me. Whenever your manager comes up to find why something that you are working on is not up, you can simply reply with something like this - If AWS with some of the smartest engineers can be down for that long do you think that our crappy service will be up 100% of time?