For anyone else expecting this to be a paper given the domain name, it’s not.
It’s a non technical interview with a couple of the original papers authors.
Not bad, just not as exciting as I imagine a paper detailing what they’ve learnt from a distributed systems perspective etc operating Dynamo then DynamoDB for so long now.
DynamoDB 10 years later
11–20 of 225 posts
Re: DynamoDB 10 years later
#12We're at early stages of planning an architecture where we offload pre-rendered JSON views of PostgreSQL onto a key value store optimised for read only high volume. Considering DynamoDB, S3, Elastic, etc. (We'll probably start without the pre-render bit, or store it in PostgreSQL until it becomes a problem). When looking at DynamoDB I noticed that there was a surprising amount of discussion around the requirement for…
There's not really magic with s3, you still need to name things with coherrent prefixes to spread around the load. DynamoDB is almost simple enough to learn in a day. And if you're doing nothing with it, you're only really paying for storage. Good luck with your decisions.
Prefixes are not needed 90% of use cases
Re: DynamoDB 10 years later
#13We tried to implement an application on DynamoDB about 2 years ago. We really struggled with implementing adhoc queries/search. For e.g:- select * from employees where name = X and city = Y. Any improvements in DynamoDB that make it easier to implement such queries?
The most reliable way to build a system with DynamoDB is to plan queries upfront. Trying to use it like a SQL database and make Ad-Hoc queries won't work because it's not a SQL DB.
Data should be stored in the fashion you wish for it to be read, and storing the same data in more than one configuration is acceptable.
Good resource: https://docs.aws.amazon.com/amazondynamodb/latest/developerg...
Re: DynamoDB 10 years later
#14We're at early stages of planning an architecture where we offload pre-rendered JSON views of PostgreSQL onto a key value store optimised for read only high volume. Considering DynamoDB, S3, Elastic, etc. (We'll probably start without the pre-render bit, or store it in PostgreSQL until it becomes a problem). When looking at DynamoDB I noticed that there was a surprising amount of discussion around the requirement for…
I have no direct experience with scaling DynamoDB in production, so take this with a grain of salt. But it seems to me that the on-demand scaling mode in DynamoDB has gotten _really_ good the last couple of years. For example, you used to have to manually set RCU/WCU to a high number when you expected a spike in traffic, since the ramp-up for on-demand scaling was pretty slow (could take up to 30 minutes). But these…
Indeed
We've some regular jobs that require scaling up dynamodb in advance few times per day, but then dynamo is only able to scale down 4x per day, so we're probably paying for over capacity unnecessarily (10x or more) for a couple hours a day
Now we just moved ondemand and let them handle it, works fine
Re: DynamoDB 10 years later
#15We tried to implement an application on DynamoDB about 2 years ago. We really struggled with implementing adhoc queries/search. For e.g:- select * from employees where name = X and city = Y. Any improvements in DynamoDB that make it easier to implement such queries?
This will sound flippant, but that's not what Dynamo is for. If you want to do freeform relational queries like that then put it in a relational database.
Dynamo is primarily designed for high volume storage/querying on well understood data sets with a few query patterns. If you want to be able to query information on employees based on their name and city you'll need to build another index keyed on name and city (in practice Dynamo makes that reasonably simple by adding a secondary index).
Re: DynamoDB 10 years later
#16If you follow Rick Houlihan (@houlihan_rick) then all the accolades that AWS for DynamoDB pale in comparison to its current team and execution in that the company seems to not be investing in it so much so that Rick left to join MongoDB.
Man I love Rick’s talks as much as anyone but let’s be real, he likely left AWS not for his love of first class geographical indexes but because Mongo offered a giant pile of money for him to evangelize their tech.
Though I have no doubts that he actually had a lot of reservations around Dynamo’s DX before, he likely has some around mongodb but those won’t be the bulk of his content
Re: DynamoDB 10 years later
#17We tried to implement an application on DynamoDB about 2 years ago. We really struggled with implementing adhoc queries/search. For e.g:- select * from employees where name = X and city = Y. Any improvements in DynamoDB that make it easier to implement such queries?
I struggled at first but I watched Advanced Design Patterns for DynamoDB[0] a few times and it clicked. As other responses have suggested, generally you define your access patterns first and then structure the data later to fit those access patterns.
Re: DynamoDB 10 years later
#18We're at early stages of planning an architecture where we offload pre-rendered JSON views of PostgreSQL onto a key value store optimised for read only high volume. Considering DynamoDB, S3, Elastic, etc. (We'll probably start without the pre-render bit, or store it in PostgreSQL until it becomes a problem). When looking at DynamoDB I noticed that there was a surprising amount of discussion around the requirement for…
Your example really summarizes the challenge with the AWS paradigm: namely that they want you to believe that the thing to do is to spread the the backend of your application across a large number of distinct data systems. No one uses DynamoDB alone: they bolt it onto Postgres after realizing they have availability or scale needs beyond what a relational database can do, then they bolt on Elasticsearch to enable querying, and then they bolt on Redis to make the disjointed backend feel fast. And I'm just talking operational use cases; ignoring analytics here. Honestly it doesn't need to be these particular technologies but this is the general phenomenon you see in so many companies that adopt a relational database, key/value store (could be Cassandra instead of DynamoDB eg like what Netflix does), a search engine, and a caching layer because they think that that's the only option
This inherently leads to a complexity debt explosion, fragmentation in the experience, and an operationally brittle posture that becomes very difficult to dig out of (this is probably why AWS loves the paradigm).
Re: DynamoDB 10 years later
#19We tried to implement an application on DynamoDB about 2 years ago. We really struggled with implementing adhoc queries/search. For e.g:- select * from employees where name = X and city = Y. Any improvements in DynamoDB that make it easier to implement such queries?
This will sound flippant, but that's not what Dynamo is for. If you want to do freeform relational queries like that then put it in a relational database. Dynamo is primarily designed for high volume storage/querying on well understood data sets with a few query patterns. If you want to be able to query information on employees based on their name and city you'll need to build another index keyed on name and city (in…
Alternatively, practice single table design: structure your table keys in such a way that they can represent all (or at least most) of the queries you need to run.
This is often easier said than done, but it can be far less expensive and more performant than adding an index for each search.
Re: DynamoDB 10 years later
#20We're at early stages of planning an architecture where we offload pre-rendered JSON views of PostgreSQL onto a key value store optimised for read only high volume. Considering DynamoDB, S3, Elastic, etc. (We'll probably start without the pre-render bit, or store it in PostgreSQL until it becomes a problem). When looking at DynamoDB I noticed that there was a surprising amount of discussion around the requirement for…
I believe it used to be static provisioning, you'd set the read and limit capacity beforehand. Then obviously there is autoscaling of those but it is still steps of capacity being provisioned. They now have a dynamic provisioning scheme, you simply don't care but it is more expensive so if you have predictible requirements it is still better to use static capacity provisioning. There is an option though. DynamoDB als…
My experience with dynamic provisioning has been that it is pretty inelastic, at least at the lower range of capacity. E.g. if you have a few read units and then try to export the data using AWS's cli client, you can pretty quickly hit the capacity limit and have to start the export over again. Last time, I ended up manually bumping the capacity way up, waiting a few minutes for the new capacity to kick in, and then exporting. Not what I had in mind when I wanted a serverless database!