Summary of the Amazon DynamoDB Service Disruption
aws.amazon.com
Summary of the Amazon DynamoDB Service Disruption
1–10 of 89 posts
Re: Summary of the Amazon DynamoDB Service Disruption
#2Unavailable servers continued to retry requests for membership data, maintaining high load on the metadata service.
It sounds like the retries exacerbated the situation. I wonder if they use exponential backoff?
Re: Summary of the Amazon DynamoDB Service Disruption
#3We did not have detailed enough monitoring for this dimension (membership size), and didn’t have enough capacity allocated to the metadata service to handle these much heavier requests.
As much as I admire and rely on AWS' scale to build architectures and fault tolerant applications, it can't be ignored that the marketing towards going "full cloud" doesn't take into account how hard it is to build resilient architectures in the cloud.
I see those disruptions events as stop signs: when the cloud itself fails to scale, I rethink a few decisions we all make when surfing those trends.
http://yourdatafitsinram.com/ also comes to mind.
Re: Summary of the Amazon DynamoDB Service Disruption
#4Unavailable servers continued to retry requests for membership data, maintaining high load on the metadata service. It sounds like the retries exacerbated the situation. I wonder if they use exponential backoff?
It's amazing how frequently this seems to happen. A lot of the huge downtime events I've read about have occurred because of failures in the systems used to recover from failures.
It seems that perhaps more time/effort needs to be spent testing the systems (and the use of those systems) which are critical for handling failure. While building them, it's easy to dismiss their scaling requirements on the basis that they shouldn't ever be under significant load.
Re: Summary of the Amazon DynamoDB Service Disruption
#5We did not have detailed enough monitoring for this dimension (membership size), and didn’t have enough capacity allocated to the metadata service to handle these much heavier requests. As much as I admire and rely on AWS' scale to build architectures and fault tolerant applications, it can't be ignored that the marketing towards going "full cloud" doesn't take into account how hard it is to build resilient architect…
How is it any harder to do in the cloud than on a rack in a warehouse? At least you don't have to muck about with cables and phoning power companies up.
Re: Summary of the Amazon DynamoDB Service Disruption
#6We did not have detailed enough monitoring for this dimension (membership size), and didn’t have enough capacity allocated to the metadata service to handle these much heavier requests. As much as I admire and rely on AWS' scale to build architectures and fault tolerant applications, it can't be ignored that the marketing towards going "full cloud" doesn't take into account how hard it is to build resilient architect…
How is it any harder to do in the cloud than on a rack in a warehouse? At least you don't have to muck about with cables and phoning power companies up.
Given they had ~300 minutes of outage in 3 years, you're looking at ~99.98% reliable in just that region. That's pretty good for a stateful serving system, and indeed you'd be pushed to do better.
Re: Summary of the Amazon DynamoDB Service Disruption
#7Unavailable servers continued to retry requests for membership data, maintaining high load on the metadata service. It sounds like the retries exacerbated the situation. I wonder if they use exponential backoff?
> I wonder if they use exponential backoff?
More importantly, did they use randomised exponential backoff?
Having all the retires hitting at the same time can lead to a pulses of outages until things settle down.
Re: Summary of the Amazon DynamoDB Service Disruption
#8We did not have detailed enough monitoring for this dimension (membership size), and didn’t have enough capacity allocated to the metadata service to handle these much heavier requests. As much as I admire and rely on AWS' scale to build architectures and fault tolerant applications, it can't be ignored that the marketing towards going "full cloud" doesn't take into account how hard it is to build resilient architect…
How is it any harder to do in the cloud than on a rack in a warehouse? At least you don't have to muck about with cables and phoning power companies up.
Just an example: during the issue even people serving 10 ops/sec, but very important 10 ops/sec, were affected by a huge complessive load which was not their for most of the part. It's true that when you "go cloud" you don't have to manage your operations, but you are basically putting everything in the hands of other op people, and what happens to you is related to a more wide set of conditions.
So managing your stuff is hard, but you are in control and can do things in a way you believe is completely safe for you. Or you at least may incur in the same events sometimes, but perhaps paying a lot less for the same services. Or you can create your deployment with characteristics which are often impossible (a lot of RAM for each server is an example) to be cost effective in the cloud.
It's not stupid to use AWS services but is not stupid to manage your operations, either in your own hardware or at least using just bare metal and/or the virtual machines service certain providers give you, but still being in part accountable, responsabile, and in control, of your system software deployment and operations.
Re: Summary of the Amazon DynamoDB Service Disruption
#9> but we should have posted the green-i to the dashboard sooner than we did
Use full orange or even half orange circle or something else if you want to convey 'subset of customers may have issues'. If possible move 'having issue' items to the top - this way people do not have to scroll down to see whats broken.
Re: Summary of the Amazon DynamoDB Service Disruption
#10Unavailable servers continued to retry requests for membership data, maintaining high load on the metadata service. It sounds like the retries exacerbated the situation. I wonder if they use exponential backoff?
It's amazing how frequently this seems to happen. A lot of the huge downtime events I've read about have occurred because of failures in the systems used to recover from failures. It seems that perhaps more time/effort needs to be spent testing the systems (and the use of those systems) which are critical for handling failure. While building them, it's easy to dismiss their scaling requirements on the basis that they…
This is a classic, even because those software parts usually are the least tested in a variety of conditions, even when they are relatively well tested. It's very hard to simulate real failures.