Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The DynamoDB Paper

brooker.co.za

Post reply on HN
Print view View original

21–30 of 95 posts

Re: The DynamoDB Paper

#21

Post by mabbo »

Rick Houlihan did a talk a few years ago about designing the data later for an application using dynamodb. The most common reaction I get from people I show it to- most of them Amazon SDEs who operate services that use Dynamodb- is "Holy shit what is this wizardry?!"

https://youtu.be/HaEPXoXVf2k

One of the biggest mistakes people make with dynamo is thinking that it's just a relational database with no relations. It's not.

It's an incredible system, but it requires a lot of deep knowledge to get the full benefits, and it requires you, often, to design your data layer very well up-front. I actually don't recommend using it for a system that hasn't mostly stabilized in design.

But when used right, it's an incredibly performant beast of a data store.

Re: The DynamoDB Paper

#22

Post by blantonl »

i_love_limes wrote: post #16
blantonl wrote: post #13

Earlier quoted context omitted.

Designing application specifically for DynamoDB will take _a lot_ of time and effort. If you can write, read, and query a JSON document using an API in your application, it's literally that simple. The only real time and effort is the architectural decisions you make up front, and that's about it. And there are some great guides out there that cover 99% of those architectural decisions. As a user of both, I find MySQ…

Have to disagree on this one. Something as basic and out of the box as a migration / data backfill is not only complicated but also very expensive (both time and cost wise) on Dynamo. Not to mention all the other things that come nicely with an relational db (type checking, auto increments, uniform data)

To be fair, the parent discusses designing an application to use Dynamo, not data migration.

I'll completely agree with you on migration / backfill. You're going to pay a lot of money to migrate a ton of data into Dynamo, and you'll also definitely increase the complexity in provisioning and setting up that migration pattern.

But my comment stands pretty well considering greefield application development around Dynamo.

Re: The DynamoDB Paper

#23

Post by [deleted] »

whalesalad wrote: post #11
ciberado wrote: post #9

Earlier quoted context omitted.

DynamoDB is amazing, but not very flexible once you have designed your database. No abstraction layer will allow you to run queries ad-hoc in a performant way.

It’s true. 400kb max item size, too. 1mb max query size I believe. Good luck grabbing a shit load of data at once without a parallel scan. Dynamo is a precision tool and it’s great at those specific workloads but it’s not a one size fits all by any means.

[deleted]

Re: The DynamoDB Paper

#24

Post by ignoramous »

ramraj07 wrote: post #18
bistablesulphur wrote: post #2

I'be been working with DynamoDB daily for a few years now, and whilst I like working with it and the specific scenario it solves for us, I'd still urge anyone thinking about using it to carefully reconsider whether their problem is truly unique enough that a traditional RDBMS couldn't handle it with some tuning. Theycan be unbelievably performant and give so much stuff for free. Designing application specifically for…

Could you elaborate on your (or a hypothetical) use case where dynamo db makes sense? I for one can never come up with something better served by rdbms or s3.

When you require point and range queries. For example, given a cart-id, fetch the skus; given a authz-token, fetch scopes; given a user-id and a time-range, fetch a list of pending order-ids.

There's a lot more you could do though, DynamoDB, is after all, a wide-column KV store. Ref this re:invent talk from 2018: https://www.youtube-nocookie.com/embed/HaEPXoXVf2k

Apart from being fully-managed, the key selling points of DynamoDB are its consistent performance for a given query type, read-your-writes consistency semantics, auto-replication, auto-disaster recovery.

See also: https://martinfowler.com/bliki/AggregateOrientedDatabase.htm... (mirror: https://archive.is/lc2eO)

Re: The DynamoDB Paper

#25

Post by abd12 »

ramraj07 wrote: post #18
bistablesulphur wrote: post #2

I'be been working with DynamoDB daily for a few years now, and whilst I like working with it and the specific scenario it solves for us, I'd still urge anyone thinking about using it to carefully reconsider whether their problem is truly unique enough that a traditional RDBMS couldn't handle it with some tuning. Theycan be unbelievably performant and give so much stuff for free. Designing application specifically for…

Could you elaborate on your (or a hypothetical) use case where dynamo db makes sense? I for one can never come up with something better served by rdbms or s3.

I always tell people there are two clear areas where DynamoDB has some major benefits:

- Very high scale applications that can be tough for an RDBMS to handle

- Serverless applications (e.g. w/ AWS Lambda) due to how the connection model (and other factors) work better with that model.

Then, for about 80% of OLTP applications, you can choose either DynamoDB or RDBMS, and it really comes down to which tradeoffs you prefer.

DynamoDB will give you consistent, predictable performance basically forever, and there's not the long-term maintenance drag of tuning your database as your usage grows. The downside, as others have mentioned, is more planning upfront and some loss of flexibility.

Re: The DynamoDB Paper

#26

Post by nkozyra »

blantonl wrote: post #13
bistablesulphur wrote: post #2

I'be been working with DynamoDB daily for a few years now, and whilst I like working with it and the specific scenario it solves for us, I'd still urge anyone thinking about using it to carefully reconsider whether their problem is truly unique enough that a traditional RDBMS couldn't handle it with some tuning. Theycan be unbelievably performant and give so much stuff for free. Designing application specifically for…

Designing application specifically for DynamoDB will take _a lot_ of time and effort. If you can write, read, and query a JSON document using an API in your application, it's literally that simple. The only real time and effort is the architectural decisions you make up front, and that's about it. And there are some great guides out there that cover 99% of those architectural decisions. As a user of both, I find MySQ…

> If you can write, read, and query a JSON document using an API in your application, it's literally that simple

You could say that of Elasticsearch or Mongo, too. And it might be technically true, but you haven't scratched the surface of mappings, design, limitations, etc.

You can dump a bunch of data into Dynamo very easily, but what about getting data via secondary indices when you can't get your data with the views you've built without scanning? How do you use partition keys in it? And so on.

Re: The DynamoDB Paper

#27

Post by blantonl »

ramraj07 wrote: post #18
bistablesulphur wrote: post #2

I'be been working with DynamoDB daily for a few years now, and whilst I like working with it and the specific scenario it solves for us, I'd still urge anyone thinking about using it to carefully reconsider whether their problem is truly unique enough that a traditional RDBMS couldn't handle it with some tuning. Theycan be unbelievably performant and give so much stuff for free. Designing application specifically for…

Could you elaborate on your (or a hypothetical) use case where dynamo db makes sense? I for one can never come up with something better served by rdbms or s3.

I'll give you two use cases that I use for DynamoDB, where otherwise I'm primarily a MySQL shop

1) Simple: I have a system that constantly records and stores 30 minute MP3 files of audio streams (1000's of them) in S3. We write the referencing metadata to a table in DynamoDB where users can query by date/time. Given the sheer amount of items (hundreds of millions), we saw far worse performance vs. cost on MySQL vs Dynamo.

2) Complex: I have a system that ingests thousands of tiny MP3 files a minute into S3 and writes the associated metadata to DynamoDB. DynamoDB then has a stream associated with it that runs a lambda to consolidate statistics to another table and stream that metadata to clients via other lambdas or data streams.

Those are two great use cases where we saw better usage patterns with Dynamo vs MySQL.

Re: The DynamoDB Paper

#28

Post by ronjouch »

mabbo wrote: post #21

Rick Houlihan did a talk a few years ago about designing the data later for an application using dynamodb. The most common reaction I get from people I show it to- most of them Amazon SDEs who operate services that use Dynamodb- is "Holy shit what is this wizardry?!" https://youtu.be/HaEPXoXVf2k One of the biggest mistakes people make with dynamo is thinking that it's just a relational database with no relations. It'…

For explicitness & searchability, commenting with the title of this talk, which is indeed excellent, not limited to DynamoDB, and which was kind of a revelation after years of using DynamoDB suboptimally:

Rick Houlihan - AWS re:Invent 2018: Amazon DynamoDB Deep Dive: Advanced Design Patterns for DynamoDB (DAT401) , https://www.youtube.com/watch?v=HaEPXoXVf2k

It should be watched along with reading the associated doc: https://docs.aws.amazon.com/amazondynamodb/latest/developerg...

Re: The DynamoDB Paper

#29

Post by time0ut »

mabbo wrote: post #21

Rick Houlihan did a talk a few years ago about designing the data later for an application using dynamodb. The most common reaction I get from people I show it to- most of them Amazon SDEs who operate services that use Dynamodb- is "Holy shit what is this wizardry?!" https://youtu.be/HaEPXoXVf2k One of the biggest mistakes people make with dynamo is thinking that it's just a relational database with no relations. It'…

I also recommend Alex DeBrie's "The DynamoDB Book" (https://www.dynamodbbook.com/). It is a great resource that talks about these design patterns in depth. It has served me and my team well over the past few years.

Re: The DynamoDB Paper

#30

Post by lumost »

blantonl wrote: post #13
bistablesulphur wrote: post #2

I'be been working with DynamoDB daily for a few years now, and whilst I like working with it and the specific scenario it solves for us, I'd still urge anyone thinking about using it to carefully reconsider whether their problem is truly unique enough that a traditional RDBMS couldn't handle it with some tuning. Theycan be unbelievably performant and give so much stuff for free. Designing application specifically for…

Designing application specifically for DynamoDB will take _a lot_ of time and effort. If you can write, read, and query a JSON document using an API in your application, it's literally that simple. The only real time and effort is the architectural decisions you make up front, and that's about it. And there are some great guides out there that cover 99% of those architectural decisions. As a user of both, I find MySQ…

It’s a question of change resilience. You can implement crud on a single object with ddb trivially. You can’t implement 5 different list by X property apis trivially, or filter the objects, or deal with foreign keys…
Post reply on HN