DynamoDB 10 years later • Hacker News Board

Re: DynamoDB 10 years later

#21

Post by willcipriano » Thu, Jan 20, 2022, 12:57 PM UTC

jon-wood wrote: post #15

victor106 wrote: post #8

We tried to implement an application on DynamoDB about 2 years ago. We really struggled with implementing adhoc queries/search. For e.g:- select * from employees where name = X and city = Y. Any improvements in DynamoDB that make it easier to implement such queries?

This will sound flippant, but that's not what Dynamo is for. If you want to do freeform relational queries like that then put it in a relational database. Dynamo is primarily designed for high volume storage/querying on well understood data sets with a few query patterns. If you want to be able to query information on employees based on their name and city you'll need to build another index keyed on name and city (in…

Amazon has a perfect use case for this. You click on a product in the search results, that url contains a UUID, that UUID is used to search Dynamo and returns an object that has all the information on the product, from that you build the page.

If what you are trying to do looks more like "Give me all the customers that live in Cuba and have spent more than $10 and have green eyes", Dynamo isn't for you. You can query that way but after you put all the work in to get it up and running, you'd probably be better off with Postgres.

Re: DynamoDB 10 years later

#22

Post by nine_k » Thu, Jan 20, 2022, 12:59 PM UTC

redwood wrote: post #18

afandian wrote: post #2

We're at early stages of planning an architecture where we offload pre-rendered JSON views of PostgreSQL onto a key value store optimised for read only high volume. Considering DynamoDB, S3, Elastic, etc. (We'll probably start without the pre-render bit, or store it in PostgreSQL until it becomes a problem). When looking at DynamoDB I noticed that there was a surprising amount of discussion around the requirement for…

Your example really summarizes the challenge with the AWS paradigm: namely that they want you to believe that the thing to do is to spread the the backend of your application across a large number of distinct data systems. No one uses DynamoDB alone: they bolt it onto Postgres after realizing they have availability or scale needs beyond what a relational database can do, then they bolt on Elasticsearch to enable quer…

If this is not the only option, what would you suggest instead? How to simplify it?

Re: DynamoDB 10 years later

#23

Post by moduspol » Thu, Jan 20, 2022, 1:03 PM UTC

augustl wrote: post #3

afandian wrote: post #2

We're at early stages of planning an architecture where we offload pre-rendered JSON views of PostgreSQL onto a key value store optimised for read only high volume. Considering DynamoDB, S3, Elastic, etc. (We'll probably start without the pre-render bit, or store it in PostgreSQL until it becomes a problem). When looking at DynamoDB I noticed that there was a surprising amount of discussion around the requirement for…

I have no direct experience with scaling DynamoDB in production, so take this with a grain of salt. But it seems to me that the on-demand scaling mode in DynamoDB has gotten _really_ good the last couple of years. For example, you used to have to manually set RCU/WCU to a high number when you expected a spike in traffic, since the ramp-up for on-demand scaling was pretty slow (could take up to 30 minutes). But these…

> The downside of on-demand is the pricing - it's more expensive if you have continuous load.

True, although you don't have to make that choice permanently. You can switch from provisioned to on demand once every 24 hours.

And you can also set up application autoscaling in provisioned mode, which'll allow you to set parameters under which it'll scale your provisioned capacity up or down for you. This doesn't require any code and works pretty well if you can accept autoscaling adjustments being made in the timeframe of a minute or two.

Re: DynamoDB 10 years later

#24

Post by moduspol » Thu, Jan 20, 2022, 1:13 PM UTC

paulgb wrote: post #20

eknkc wrote: post #5

Earlier quoted context omitted.

I believe it used to be static provisioning, you'd set the read and limit capacity beforehand. Then obviously there is autoscaling of those but it is still steps of capacity being provisioned. They now have a dynamic provisioning scheme, you simply don't care but it is more expensive so if you have predictible requirements it is still better to use static capacity provisioning. There is an option though. DynamoDB als…

My experience with dynamic provisioning has been that it is pretty inelastic, at least at the lower range of capacity. E.g. if you have a few read units and then try to export the data using AWS's cli client, you can pretty quickly hit the capacity limit and have to start the export over again. Last time, I ended up manually bumping the capacity way up, waiting a few minutes for the new capacity to kick in, and then…

I understand it's not really your point, but if you're actually looking to export all the data from the table, they've got an API call you can give to have DynamoDB write the whole table to S3. This doesn't use any of your available capacity.

https://docs.aws.amazon.com/amazondynamodb/latest/developerg...

Beyond that, though, it's really not designed for that kind of use case.

Re: DynamoDB 10 years later

#25

Post by itsmemattchung » Thu, Jan 20, 2022, 1:14 PM UTC

victor106 wrote: post #8

We tried to implement an application on DynamoDB about 2 years ago. We really struggled with implementing adhoc queries/search. For e.g:- select * from employees where name = X and city = Y. Any improvements in DynamoDB that make it easier to implement such queries?

With DynamoDB, you can now execute SQL queries using PartiQL:

https://docs.aws.amazon.com/amazondynamodb/latest/developerg...

Re: DynamoDB 10 years later

#26

Post by aneil » Thu, Jan 20, 2022, 1:14 PM UTC

afandian wrote: post #2

We're at early stages of planning an architecture where we offload pre-rendered JSON views of PostgreSQL onto a key value store optimised for read only high volume. Considering DynamoDB, S3, Elastic, etc. (We'll probably start without the pre-render bit, or store it in PostgreSQL until it becomes a problem). When looking at DynamoDB I noticed that there was a surprising amount of discussion around the requirement for…

Exactly my experience. I got sucked into using more than once, thinking it would be better next time, but there are just so many sharp edges.

At one company, someone accidentally set the write rate rate high to transfer data into the db. This had the effect of permanently increasing the shard count to a huge number, basically making the DB useless.

Re: DynamoDB 10 years later

#27

Post by snorkel » Thu, Jan 20, 2022, 1:17 PM UTC

afandian wrote: post #2

We're at early stages of planning an architecture where we offload pre-rendered JSON views of PostgreSQL onto a key value store optimised for read only high volume. Considering DynamoDB, S3, Elastic, etc. (We'll probably start without the pre-render bit, or store it in PostgreSQL until it becomes a problem). When looking at DynamoDB I noticed that there was a surprising amount of discussion around the requirement for…

If you don't need data persistence then consider redis instead (which can also do persistence if you enable AOF)

Re: DynamoDB 10 years later

#28

Post by gigatexal » Thu, Jan 20, 2022, 1:24 PM UTC

vslira wrote: post #16

gigatexal wrote: post #10

If you follow Rick Houlihan (@houlihan_rick) then all the accolades that AWS for DynamoDB pale in comparison to its current team and execution in that the company seems to not be investing in it so much so that Rick left to join MongoDB.

Man I love Rick’s talks as much as anyone but let’s be real, he likely left AWS not for his love of first class geographical indexes but because Mongo offered a giant pile of money for him to evangelize their tech. Though I have no doubts that he actually had a lot of reservations around Dynamo’s DX before, he likely has some around mongodb but those won’t be the bulk of his content

At his rank at AWS I don’t know if money was such an issue. He strikes me as a person who cares deeply about the underlying tech. But I have no idea one way or the other.

Re: DynamoDB 10 years later

#29

Post by ellimilial » Thu, Jan 20, 2022, 1:25 PM UTC

owenmarshall wrote: post #19

jon-wood wrote: post #15

Earlier quoted context omitted.

This will sound flippant, but that's not what Dynamo is for. If you want to do freeform relational queries like that then put it in a relational database. Dynamo is primarily designed for high volume storage/querying on well understood data sets with a few query patterns. If you want to be able to query information on employees based on their name and city you'll need to build another index keyed on name and city (in…

Alternatively, practice single table design: structure your table keys in such a way that they can represent all (or at least most) of the queries you need to run. This is often easier said than done, but it can be far less expensive and more performant than adding an index for each search.

It's always great fun compounding new, manual 'indexes' when you discover you need another query.

Re: DynamoDB 10 years later

#30

Post by twodayrice » Thu, Jan 20, 2022, 1:26 PM UTC

afandian wrote: post #2

We're at early stages of planning an architecture where we offload pre-rendered JSON views of PostgreSQL onto a key value store optimised for read only high volume. Considering DynamoDB, S3, Elastic, etc. (We'll probably start without the pre-render bit, or store it in PostgreSQL until it becomes a problem). When looking at DynamoDB I noticed that there was a surprising amount of discussion around the requirement for…

I think this is a good summary, and it even gets more complicated if you start using the DAX cache. Your read/write provisioning for DAX is totally different than the underlying dynamodb tables. The write throughput for Dax is limited by the size of the master node in the cluster. Can you say bottleneck?