11–20 of 89 posts
"For us, availability is the most important feature of DynamoDB"
I would think that durability should be the most important feature of DynamoDB; better to have intermittent outage, or reduced capacity, but not have data loss - but perhaps there is something about DynamoDB which suggests availability is rated more highly than durability?
We did not have detailed enough monitoring for this dimension (membership size), and didn’t have enough capacity allocated to the metadata service to handle these much heavier requests. As much as I admire and rely on AWS' scale to build architectures and fault tolerant applications, it can't be ignored that the marketing towards going "full cloud" doesn't take into account how hard it is to build resilient architect…
that said, even with those disruptions and whatnot happening on Amazon as a warning, I am not skilled enough nor have time enough to build a non cloud resilient infrastructure.
I was looking to go with redundant vps at first, because amazon does have high cost for us, however, just learning all the things that can go wrong in the first very part, the load balancer, and all the gritty details one have to consider for just this little component to support interruption free failover, made me rethink the cost benefit of going managed.
it is true that going cloud doesn't really remove outages risks completely and it will not be as resilient as an infrastructure built with skill and love by the best out there, but how many shops can actually roll with their own solution and get an equivalent level of availability?
scaling web nodes is within my capabilities, building a ha database is already quite above my skill but I may manage, testing database failover, making sure it works, making sure that it can actually recover from one node dying and that the application stay live meanwhile? that's way above what I can reasonably do and what my company can afford to pay maintenance for.
A pretty good post mortem, but one worrisome final comment: "For us, availability is the most important feature of DynamoDB" I would think that durability should be the most important feature of DynamoDB; better to have intermittent outage, or reduced capacity, but not have data loss - but perhaps there is something about DynamoDB which suggests availability is rated more highly than durability?
A pretty good post mortem, but one worrisome final comment: "For us, availability is the most important feature of DynamoDB" I would think that durability should be the most important feature of DynamoDB; better to have intermittent outage, or reduced capacity, but not have data loss - but perhaps there is something about DynamoDB which suggests availability is rated more highly than durability?
I personally didnt know they used meta data tables in Dynamo, and that sounds like they lost one of the most important traits which gave Dynamo its super-high availability abilities, which is having all the cluster state in the clients rather than in the servers.
A pretty good post mortem, but one worrisome final comment: "For us, availability is the most important feature of DynamoDB" I would think that durability should be the most important feature of DynamoDB; better to have intermittent outage, or reduced capacity, but not have data loss - but perhaps there is something about DynamoDB which suggests availability is rated more highly than durability?
Looks like a fair number of key aws systems rely on DynamoDB -- and further, the same system used by customers. I wonder: do they have any inclination to decouple these dependencies to prevent correlated outages?
A pretty good post mortem, but one worrisome final comment: "For us, availability is the most important feature of DynamoDB" I would think that durability should be the most important feature of DynamoDB; better to have intermittent outage, or reduced capacity, but not have data loss - but perhaps there is something about DynamoDB which suggests availability is rated more highly than durability?
There are many things to like about DynamoDB, but one of the things I really dislike is the 'abscence of an error == success' pattern that they implement for many operations. With less frequent metadata requests, I imagine there is a greater chance of silent failures (ie availability looks fine, durability is gone).
Could be wrong though, I'm not going to pretend to know more about distributed systems than the DynamoDB developers.
Would it have been wiser for Amazon to implement a completely separate instance of DynamoDB for service offerings that depended on it? Or is this just simply cost ineffective? Help me understand, thanks. =)
Looks like a fair number of key aws systems rely on DynamoDB -- and further, the same system used by customers. I wonder: do they have any inclination to decouple these dependencies to prevent correlated outages?
I don't think they rely on Dynamo, they rely on an internal metadata service that Dynamo just happened to overwhelm with too many large requests.
> ... subsequent impact to other AWS services that depend on DynamoDB ...
And the metadata service is part of DynamoDB:
> The membership of a set of table/partitions within a server is managed by DynamoDB’s internal metadata service.