Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

DynamoDB 10 years later

amazon.science

Post reply on HN
Print view View original

101–110 of 225 posts

Re: DynamoDB 10 years later

#101

Post by didip »

To be honest, as a customer, it is hard for me to justify using DynamoDB. Some of this criticism can be out of date:

1. DynamoDB is not as convenient. There are a bit too many dials to turn.

2. DynamoDB does not have a SQL facade on top.

3. DynamoDB is proprietary, I believe there's no OSS API equivalent if you want to migrate out.

4. DynamoDB was kind of expensive. But it has been a while since I last check the pricing page.

It's simply much better to start with PostgreSQL Aurora and move to a more scalable storage based on specific uses-cases later. For example: Cassandra, Elastic, Druid, or CockroachDB.

Re: DynamoDB 10 years later

#102

Post by uberdru »

Marazan wrote: post #39
afandian wrote: post #2

We're at early stages of planning an architecture where we offload pre-rendered JSON views of PostgreSQL onto a key value store optimised for read only high volume. Considering DynamoDB, S3, Elastic, etc. (We'll probably start without the pre-render bit, or store it in PostgreSQL until it becomes a problem). When looking at DynamoDB I noticed that there was a surprising amount of discussion around the requirement for…

DyanmoDB is pretty much the opposite of magic. It is a resource that can often be the right tool for the job but you really have to understand what the job is and carefully measure Dynamo up for what you are doing. It is _easy_ to misunderstand or miss something that would make Dynamo hideously expensive for your use case.

What use cases would likely make it hideously expensive, in your view? Like, what are the red flags?

Re: DynamoDB 10 years later

#103

Post by nathanfig »

didip wrote: post #101

To be honest, as a customer, it is hard for me to justify using DynamoDB. Some of this criticism can be out of date: 1. DynamoDB is not as convenient. There are a bit too many dials to turn. 2. DynamoDB does not have a SQL facade on top. 3. DynamoDB is proprietary, I believe there's no OSS API equivalent if you want to migrate out. 4. DynamoDB was kind of expensive. But it has been a while since I last check the pric…

That’s surprising to me, I consider DynamoDB to be far simpler than any relational DB including the alternatives you list

Re: DynamoDB 10 years later

#104

Post by eropple »

picardo wrote: post #46
ndm000 wrote: post #34

Earlier quoted context omitted.

> they bolt it onto Postgres I am working with a company that is redesigning an enterprise transactional system, currently backed by an Oracle database with 3000 tables. It’s B2B so loads are predictable and are expected to grow no more than 10% per year. They want to use DynamoDB as their primary data store, with Postgres for edge cases it seems to me the opposite would be more beneficial. At what point does DynamoD…

You can make Postgres scale, but there is an operational cost to it. DynamoDB does that for you out of the box. (So does Aurora, to be honest, but there is also an overhead to setting up an Aurora cluster to the needs of your business.) I've found also that in Postgres the query performance does not keep up with bursts of traffic -- you need to overprovision your db servers to cope with the highest traffic days. Dyna…

> So DynamoDB is pretty great. But to get the the best experience from DynamoDB, you need to have a stable codebase, and design your tables around your access patterns. Because joining two tables isn't fun.

More than just joining--you're in the unenviable place of reinventing (in most environments, anyway) a lot of what are just online problems in the SQL universe. Stuff you'd do with a case statement in Postgres becomes some on-the-worker shenanigans, stuff you'd do with a materialized view in Postgres becomes a batch process that itself has to be babysat and managed and introduces new and exciting flavors of contention.

There are really good reasons to use DynamoDB out there, but there are also an absolute ton of land mines. If your data model isn't trivial, DynamoDB's best use case is in making faster subsets of your data model that you can make trivial.

Re: DynamoDB 10 years later

#105

Post by 300bps »

jamesblonde wrote: post #93
300bps wrote: post #61

Earlier quoted context omitted.

In my experience, leave DynamoDB alone and it will work great. My experience agrees with yours and I'm likewise puzzled by the grandparent comment. But just a shout out to DAX (DyanmoDB Accelerator) which makes it scale through the roof: https://aws.amazon.com/dynamodb/dax/

If you add DAX you are not guaranteed to read your writes. Terrible consistency model. https://docs.aws.amazon.com/amazondynamodb/latest/developerg...

Terrible consistency model.

Judging a consistency model as "terrible" implies that it does not fit any use case and therefore is objectively bad.

On the contrary, there are plenty of use cases where "eventually consistent writes" is the perfect use case. To judge this as true, you only have to look and see that every major database server offers this as an option - just one example:

https://www.compose.com/articles/postgresql-and-per-connecti...

Re: DynamoDB 10 years later

#107

Post by rmbyrro »

jamesblonde wrote: post #93
300bps wrote: post #61

Earlier quoted context omitted.

In my experience, leave DynamoDB alone and it will work great. My experience agrees with yours and I'm likewise puzzled by the grandparent comment. But just a shout out to DAX (DyanmoDB Accelerator) which makes it scale through the roof: https://aws.amazon.com/dynamodb/dax/

If you add DAX you are not guaranteed to read your writes. Terrible consistency model. https://docs.aws.amazon.com/amazondynamodb/latest/developerg...

I think main advantage of DDB is being serverless. Adding a server-based layer on top of it doesn't make sense to me.

I have a theory it would be better to have multiple table-replicas for read access. At application level, you randomize access to those tables according to your read scale needs.

Use main table streams and lambda to keep replicas in sync.

Depending on your traffic, this might end more expensive than DAX, but you remain fully serverless, using the exact same technology model, and have control over the consistency model.

Haven't had the chance to test this in practice, though.

Re: DynamoDB 10 years later

#108

Post by phamilton »

afandian wrote: post #2

We're at early stages of planning an architecture where we offload pre-rendered JSON views of PostgreSQL onto a key value store optimised for read only high volume. Considering DynamoDB, S3, Elastic, etc. (We'll probably start without the pre-render bit, or store it in PostgreSQL until it becomes a problem). When looking at DynamoDB I noticed that there was a surprising amount of discussion around the requirement for…

I don't know your scaling needs, but I would highly recommend just using Aurora postgresql for read-only workloads. We have some workloads that are essentially K/V store lookups that were previously slated for dynamodb. On an Aurora cluster of 3*r6g.xlarge we easily handle 25k qps with p99 in the single-digit ms range. Aurora can scale up to 15 instances and up to 24xlarge, so it would not be unreasonable to see 100x the read workload with similar latencies.

Happy to talk more. We're actively moving a bunch of workloads away from DynamoDB and to Aurora so this is fresh on our minds.

Re: DynamoDB 10 years later

#109

Post by Marazan »

uberdru wrote: post #102
Marazan wrote: post #39

Earlier quoted context omitted.

DyanmoDB is pretty much the opposite of magic. It is a resource that can often be the right tool for the job but you really have to understand what the job is and carefully measure Dynamo up for what you are doing. It is _easy_ to misunderstand or miss something that would make Dynamo hideously expensive for your use case.

What use cases would likely make it hideously expensive, in your view? Like, what are the red flags?

Hot keys are the primary one. They destroys your "average" calculations for your throughput.

Bulk loading data is the other gotcha I've run into. Had a beautiful use case for steady read performance of a batch dataset that was incredibly economical on Dynamo but the cost/time for loading the dataset into Dynamo was totally prohibitive.

Basically Dynamo is great for constant read/write of very small, randomly distributed documents. Once you are out of thay zone things can hey dicey fast.

Re: DynamoDB 10 years later

#110

Post by geodel »

salil999 wrote: post #65

I can safely say that the team members working in DynamoDB are very skilled and they care deeply about the product. They really work hard and think of interesting solutions to a lot of problems that their biggest customers face which is great from a product standpoint. There are some pretty smart people working there. Engineering, however, was a disaster story. Code is horribly written and very few tests are maintain…

Very interesting!

To me the takeaway is large/interesting/challenging engineering projects are pretty close to disasters generally. Some time they do become disaster actually.

On the other hand if a project looks like straight up designed, neatly put into JIRA stories, and developers deliver code consistently week after week then it may be a successfully planned and delivered project. But it would mostly be doing stuff that has already been many times over and likely by same people on team.

At least this has been my experience while working on standardized / templated projects vs something new.

Post reply on HN