DynamoDB 10 years later
amazon.science
DynamoDB 10 years later
1–10 of 225 posts
Re: DynamoDB 10 years later
#2We're at early stages of planning an architecture where we offload pre-rendered JSON views of PostgreSQL onto a key value store optimised for read only high volume. Considering DynamoDB, S3, Elastic, etc. (We'll probably start without the pre-render bit, or store it in PostgreSQL until it becomes a problem).
When looking at DynamoDB I noticed that there was a surprising amount of discussion around the requirement for provisioning, considering node read/write ratios, data characteristics, etc. Basically, worrying about all the stuff you'd have to worry about with a traditional database.
To be honest, I'd hoped that it could be a bit more 'magic', like S3, and it AWS would take care of provisioning, scaling, sharding etc. But it seemed disappointingly that you'd have to focus on proactively worrying about operations and provisioning.
Is that sense correct? Is the dream of a self-managing, fire-and-forget key value database completely naive?
Re: DynamoDB 10 years later
#3We're at early stages of planning an architecture where we offload pre-rendered JSON views of PostgreSQL onto a key value store optimised for read only high volume. Considering DynamoDB, S3, Elastic, etc. (We'll probably start without the pre-render bit, or store it in PostgreSQL until it becomes a problem). When looking at DynamoDB I noticed that there was a surprising amount of discussion around the requirement for…
I have no direct experience with scaling DynamoDB in production, so take this with a grain of salt. But it seems to me that the on-demand scaling mode in DynamoDB has gotten _really_ good the last couple of years.
For example, you used to have to manually set RCU/WCU to a high number when you expected a spike in traffic, since the ramp-up for on-demand scaling was pretty slow (could take up to 30 minutes). But these days, on-demand can handle spikes from 10s of requests a minute to 100s/1000s per second gracefully.
The downside of on-demand is the pricing - it's more expensive if you have continuous load. But it can easily become _much_ cheaper if you have naturally spiky load patterns.
Example: https://aws.amazon.com/blogs/database/running-spiky-workload...
Re: DynamoDB 10 years later
#4We're at early stages of planning an architecture where we offload pre-rendered JSON views of PostgreSQL onto a key value store optimised for read only high volume. Considering DynamoDB, S3, Elastic, etc. (We'll probably start without the pre-render bit, or store it in PostgreSQL until it becomes a problem). When looking at DynamoDB I noticed that there was a surprising amount of discussion around the requirement for…
It is for now but it doesn't have to be. Dynamo's design isn't particularly amenable to dynamic and heterogenous shard topologies however.
There could exist a fantasy database where you still tell it your hash and range keys, which are roughly how you tell the database which data isn't closely related to each other and which data is (and which you may want to scan) but instead of hard provisioning shard capacity it automagically splits shards when they hotspot and doesn't rely consistent hashing so that every shard can be sized differently depending on how hot it is.
Right now such a database doesn't exist AFAICT as most places that need something the scales big enough also generally have the skill to avoid most of the pitfalls that cause problems on simple databases like Dynamo.
Re: DynamoDB 10 years later
#5We're at early stages of planning an architecture where we offload pre-rendered JSON views of PostgreSQL onto a key value store optimised for read only high volume. Considering DynamoDB, S3, Elastic, etc. (We'll probably start without the pre-render bit, or store it in PostgreSQL until it becomes a problem). When looking at DynamoDB I noticed that there was a surprising amount of discussion around the requirement for…
I believe it used to be static provisioning, you'd set the read and limit capacity beforehand. Then obviously there is autoscaling of those but it is still steps of capacity being provisioned.
They now have a dynamic provisioning scheme, you simply don't care but it is more expensive so if you have predictible requirements it is still better to use static capacity provisioning. There is an option though.
DynamoDB also requires the developer to know about its data storage model. While this is generally a good practice for any data storage solution, I feel like Dynamo requires a lot more careful planning.
I also think that most of the best practices, articles etc apply to giant datasets with huge scale issues etc. If you are running a moderately active app, you probably can get away with a lot of stupid design decisions.
Re: DynamoDB 10 years later
#6We're at early stages of planning an architecture where we offload pre-rendered JSON views of PostgreSQL onto a key value store optimised for read only high volume. Considering DynamoDB, S3, Elastic, etc. (We'll probably start without the pre-render bit, or store it in PostgreSQL until it becomes a problem). When looking at DynamoDB I noticed that there was a surprising amount of discussion around the requirement for…
After looking into solutions like Fauna, Upstash, and Planetscale I don't understand why anyone is bothering with DDB anymore.
I read "the dynamodb book" and almost got a stroke. So much idiosyncrasies, for what?!
Re: DynamoDB 10 years later
#7We're at early stages of planning an architecture where we offload pre-rendered JSON views of PostgreSQL onto a key value store optimised for read only high volume. Considering DynamoDB, S3, Elastic, etc. (We'll probably start without the pre-render bit, or store it in PostgreSQL until it becomes a problem). When looking at DynamoDB I noticed that there was a surprising amount of discussion around the requirement for…
Plenty of options already exist. DynamoDB has both autoscaling and serverless modes. AWS also has managed Cassandra (runs on top of DynamoDB) which doesn't need instance management.
Azure has CosmosDB, GCP has Cloud Datastore/Firestore, and there are many DB vendors like Planetscale (mysql), CockroachDB (postgres), FaunaDB (custom document/relational) that have "serverless" options.
Re: DynamoDB 10 years later
#8We tried to implement an application on DynamoDB about 2 years ago.
We really struggled with implementing adhoc queries/search. For e.g:- select * from employees where name = X and city = Y.
Any improvements in DynamoDB that make it easier to implement such queries?
Re: DynamoDB 10 years later
#9We're at early stages of planning an architecture where we offload pre-rendered JSON views of PostgreSQL onto a key value store optimised for read only high volume. Considering DynamoDB, S3, Elastic, etc. (We'll probably start without the pre-render bit, or store it in PostgreSQL until it becomes a problem). When looking at DynamoDB I noticed that there was a surprising amount of discussion around the requirement for…
There's not really magic with s3, you still need to name things with coherrent prefixes to spread around the load.
DynamoDB is almost simple enough to learn in a day. And if you're doing nothing with it, you're only really paying for storage. Good luck with your decisions.
Re: DynamoDB 10 years later
#10If you follow Rick Houlihan (@houlihan_rick) then all the accolades that AWS for DynamoDB pale in comparison to its current team and execution in that the company seems to not be investing in it so much so that Rick left to join MongoDB.