Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

DynamoDB 10 years later

amazon.science

Post reply on HN
Print view View original

31–40 of 225 posts

Re: DynamoDB 10 years later

#31

Post by manigandham »

victor106 wrote: post #8

We tried to implement an application on DynamoDB about 2 years ago. We really struggled with implementing adhoc queries/search. For e.g:- select * from employees where name = X and city = Y. Any improvements in DynamoDB that make it easier to implement such queries?

DynamoDB (and other dynamo-like systems like Cassandra, Bigtable) are just advanced key/value stores. They support multiple levels of keys->values but fundamentally you need the key to find the associated value.

If you want to search by parameters that aren't keys then you need to store your data that way. Most of these systems have secondary indexes now, and that's basically what they do for you automatically in the backend, storing another copy of your records using a different key.

If you need adhoc relational queries then you should use a relational database.

Re: DynamoDB 10 years later

#32

Post by augustl »

itsmemattchung wrote: post #25
victor106 wrote: post #8

We tried to implement an application on DynamoDB about 2 years ago. We really struggled with implementing adhoc queries/search. For e.g:- select * from employees where name = X and city = Y. Any improvements in DynamoDB that make it easier to implement such queries?

With DynamoDB, you can now execute SQL queries using PartiQL: https://docs.aws.amazon.com/amazondynamodb/latest/developerg...

Note that this is just a new syntax for the existing querying capabilities. If you query something that's not in the hash/sort key, you still need to filter on the "client" after the 1mb data set size limit etc.

Re: DynamoDB 10 years later

#33

Post by Jupe »

victor106 wrote: post #8

We tried to implement an application on DynamoDB about 2 years ago. We really struggled with implementing adhoc queries/search. For e.g:- select * from employees where name = X and city = Y. Any improvements in DynamoDB that make it easier to implement such queries?

DynamoDB is not meant for ad-hoc query patterns; as others have said, plan your indexes around your access patterns.

However, so long as you add a global secondary index (GSI) with name, city as the key, you can certainly do such things. But be aware for large-scale solutions:

1. There's a limit of 20 GSIs per table. You can increase with a call to AWS support.

2. GSIs are latently updated; read-after write is not guaranteed, and there is no "consistent read" option on a GSI like there is with tables.

3. WCUs on GSIs should match (or surpass) the WCUs on the original table, else throughput limit exceeded exceptions will occur. So, 3 GSIs on a table means you pay 4x+ in WCU costs.

4. The keys of the GSI should be evenly distributed, just like the PK on a main table. If not, there is additional opportunity for hot partitions on write.

Ref: https://aws.amazon.com/premiumsupport/knowledge-center/dynam...

Re: DynamoDB 10 years later

#34

Post by ndm000 »

redwood wrote: post #18
afandian wrote: post #2

We're at early stages of planning an architecture where we offload pre-rendered JSON views of PostgreSQL onto a key value store optimised for read only high volume. Considering DynamoDB, S3, Elastic, etc. (We'll probably start without the pre-render bit, or store it in PostgreSQL until it becomes a problem). When looking at DynamoDB I noticed that there was a surprising amount of discussion around the requirement for…

Your example really summarizes the challenge with the AWS paradigm: namely that they want you to believe that the thing to do is to spread the the backend of your application across a large number of distinct data systems. No one uses DynamoDB alone: they bolt it onto Postgres after realizing they have availability or scale needs beyond what a relational database can do, then they bolt on Elasticsearch to enable quer…

> they bolt it onto Postgres

I am working with a company that is redesigning an enterprise transactional system, currently backed by an Oracle database with 3000 tables. It’s B2B so loads are predictable and are expected to grow no more than 10% per year.

They want to use DynamoDB as their primary data store, with Postgres for edge cases it seems to me the opposite would be more beneficial.

At what point does DynamoDB become a better choice than Postgres? I know that at certain scales Postgres breaks down, but what are those thresholds?

Re: DynamoDB 10 years later

#35

Post by [deleted] »

victor106 wrote: post #8

We tried to implement an application on DynamoDB about 2 years ago. We really struggled with implementing adhoc queries/search. For e.g:- select * from employees where name = X and city = Y. Any improvements in DynamoDB that make it easier to implement such queries?

[deleted]

Re: DynamoDB 10 years later

#36

Post by brodouevencode »

PaywallBuster wrote: post #12
gonzo41 wrote: post #9

Earlier quoted context omitted.

There's not really magic with s3, you still need to name things with coherrent prefixes to spread around the load. DynamoDB is almost simple enough to learn in a day. And if you're doing nothing with it, you're only really paying for storage. Good luck with your decisions.

Prefixes are not needed 90% of use cases

I'm not going to speculate on the accuracy of 90% value, but I will say that appropriately prefixed objects substantially help with performance when you have tons of small-ish files. Maybe most orgs don't have that need but in operational realms doing this with your logs make the response faster.

Re: DynamoDB 10 years later

#37

Post by PaywallBuster »

moduspol wrote: post #23
augustl wrote: post #3

Earlier quoted context omitted.

I have no direct experience with scaling DynamoDB in production, so take this with a grain of salt. But it seems to me that the on-demand scaling mode in DynamoDB has gotten _really_ good the last couple of years. For example, you used to have to manually set RCU/WCU to a high number when you expected a spike in traffic, since the ramp-up for on-demand scaling was pretty slow (could take up to 30 minutes). But these…

> The downside of on-demand is the pricing - it's more expensive if you have continuous load. True, although you don't have to make that choice permanently. You can switch from provisioned to on demand once every 24 hours. And you can also set up application autoscaling in provisioned mode, which'll allow you to set parameters under which it'll scale your provisioned capacity up or down for you. This doesn't require…

scaling down is limited to 4x a day

Re: DynamoDB 10 years later

#38

Post by vosper »

ndm000 wrote: post #34
redwood wrote: post #18

Earlier quoted context omitted.

Your example really summarizes the challenge with the AWS paradigm: namely that they want you to believe that the thing to do is to spread the the backend of your application across a large number of distinct data systems. No one uses DynamoDB alone: they bolt it onto Postgres after realizing they have availability or scale needs beyond what a relational database can do, then they bolt on Elasticsearch to enable quer…

> they bolt it onto Postgres I am working with a company that is redesigning an enterprise transactional system, currently backed by an Oracle database with 3000 tables. It’s B2B so loads are predictable and are expected to grow no more than 10% per year. They want to use DynamoDB as their primary data store, with Postgres for edge cases it seems to me the opposite would be more beneficial. At what point does DynamoD…

They should be looking at Aurora, not Dynamo. Using Dynamo as the primary store for relational data (3000 tables!) sounds like an awful idea to me. I’d rather stay on Oracle.

https://aws.amazon.com/rds/aurora/?aurora-whats-new.sort-by=...

Re: DynamoDB 10 years later

#39

Post by Marazan »

afandian wrote: post #2

We're at early stages of planning an architecture where we offload pre-rendered JSON views of PostgreSQL onto a key value store optimised for read only high volume. Considering DynamoDB, S3, Elastic, etc. (We'll probably start without the pre-render bit, or store it in PostgreSQL until it becomes a problem). When looking at DynamoDB I noticed that there was a surprising amount of discussion around the requirement for…

DyanmoDB is pretty much the opposite of magic.

It is a resource that can often be the right tool for the job but you really have to understand what the job is and carefully measure Dynamo up for what you are doing.

It is _easy_ to misunderstand or miss something that would make Dynamo hideously expensive for your use case.

Post reply on HN