Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The DynamoDB Paper

brooker.co.za

Post reply on HN
Print view View original

71–80 of 95 posts

Re: The DynamoDB Paper

#71

Post by lliamander »

bistablesulphur wrote: post #2

I'be been working with DynamoDB daily for a few years now, and whilst I like working with it and the specific scenario it solves for us, I'd still urge anyone thinking about using it to carefully reconsider whether their problem is truly unique enough that a traditional RDBMS couldn't handle it with some tuning. Theycan be unbelievably performant and give so much stuff for free. Designing application specifically for…

I think there are good reasons to choose DynamoDB over a RDBMS that have nothing to do with scalability.

I've used DynamoDB several times over the past several years in the context of providing a datastore for a microservice. In all cases it was cheaper and easier than RDS, and the ability to add GSIs has enabled me to adapt to all of the new access patterns I've had to deal with.

For us, DynamoDB has become a 'boring' option.

Re: The DynamoDB Paper

#72

Post by mikhmha »

ledauphin wrote: post #54

An underrated part of DynamoDB are its streams. You can subscribe to changes and reliably process those in a distributed way. If you're comfortable with the terms "at-least once delivery" and "eventual consistency", you can build some truly amazing systems by letting events propagate reactively through your system, never touching a data store or messaging broker other than DynamoDB itself. It's not for everyone, but…

Yeah we make use of streams at my work. Really useful. You can hook up streams to a Lambda and have it process events and flow them downstream to a Data Lake or Data Warehouse for analytic workloads. What works really well is pushing data to an S3 bucket with object versioning and replication enabled.

I think dynamoDB streams and kinesis streams work similar under the hood? But dynamoDB streams are way cheaper, pricing is on-demand compared to hourly for Kinesis.

Re: The DynamoDB Paper

#73

Post by ramraj07 »

ignoramous wrote: post #24
ramraj07 wrote: post #18

Earlier quoted context omitted.

Could you elaborate on your (or a hypothetical) use case where dynamo db makes sense? I for one can never come up with something better served by rdbms or s3.

When you require point and range queries. For example, given a cart-id, fetch the skus; given a authz-token, fetch scopes; given a user-id and a time-range, fetch a list of pending order-ids. There's a lot more you could do though, DynamoDB, is after all, a wide-column KV store. Ref this re:invent talk from 2018: https://www.youtube-nocookie.com/embed/HaEPXoXVf2k Apart from being fully-managed, the key selling points…

The aws reinvent lecture was great and answered exactly when to use dynamodb. I might seriously consider it for some of my applications for sure.

Re: The DynamoDB Paper

#74

Post by LAC-Tech »

mabbo wrote: post #21

Rick Houlihan did a talk a few years ago about designing the data later for an application using dynamodb. The most common reaction I get from people I show it to- most of them Amazon SDEs who operate services that use Dynamodb- is "Holy shit what is this wizardry?!" https://youtu.be/HaEPXoXVf2k One of the biggest mistakes people make with dynamo is thinking that it's just a relational database with no relations. It'…

Thanks, bookmarked this. It's good to see a proper take on data modelling on document stores instead of just "through any old JSON in there it'll be fine!!!"

Re: The DynamoDB Paper

#75

Post by Twirrim »

davidjfelix wrote: post #46
mabbo wrote: post #21

Rick Houlihan did a talk a few years ago about designing the data later for an application using dynamodb. The most common reaction I get from people I show it to- most of them Amazon SDEs who operate services that use Dynamodb- is "Holy shit what is this wizardry?!" https://youtu.be/HaEPXoXVf2k One of the biggest mistakes people make with dynamo is thinking that it's just a relational database with no relations. It'…

It's worth noting that a lot of the early database designs, including this 2018 video pre-date some dramatic improvements to dynamodb usability. I think the biggest ones were: - an increase in the number of GSIs you can create (Dec 2018) [1] - making on-demand possible [2] - an increase in the default limit for number of tables you can create (Mar 2022) [3] I don't think these new features necessarily make the single…

Something else important to mention is that dynamodb now re-consolidates tables.

This is a lousy explanation, but Read/Write quota is split evenly over all partitions. Each partition is created based on the hash-key used, and there's an upper limit on how much data can be stored in any given partition. So if you end up with a hot hash-key, lots of stuff in it, that data gets split over more and more and more partitions, and the overall throughput goes down (quota is split evenly over partitions).

I believe this is still a general risk, and you need to be extremely canny about your use of hash key to avoid it, but historically they couldn't reconsolidate partitions. So you'd end up with a table in a terrible state with quota having to be sky high to still get effective performance. The only option then was to completely rotate tables. New table with a better hash-key, migrate data (or whatever else you needed to do).

Now at least, once the data is gone, the partitions will reconsolidate, so an entire table isn't a complete loss.

Re: The DynamoDB Paper

#76

Post by pdhborges »

davidjfelix wrote: post #46
mabbo wrote: post #21

Rick Houlihan did a talk a few years ago about designing the data later for an application using dynamodb. The most common reaction I get from people I show it to- most of them Amazon SDEs who operate services that use Dynamodb- is "Holy shit what is this wizardry?!" https://youtu.be/HaEPXoXVf2k One of the biggest mistakes people make with dynamo is thinking that it's just a relational database with no relations. It'…

It's worth noting that a lot of the early database designs, including this 2018 video pre-date some dramatic improvements to dynamodb usability. I think the biggest ones were: - an increase in the number of GSIs you can create (Dec 2018) [1] - making on-demand possible [2] - an increase in the default limit for number of tables you can create (Mar 2022) [3] I don't think these new features necessarily make the single…

People don't need to be scared they just need to do their homework.

In my opinion having more tables and more GSIs available won't help you very much if you started with flawed data model (unless you kept making the same design mistakes 256 times). A team that tries to claw back from a flawed table design by pilling up GSIs is just in for a world of pain.

So if you are planing to go with Dynamo: - Read about the data modeling tecniques - Figure out your access patterns - Check if your application and model can withstand the eventual consistency of GSIs - Have a plan to rework your data model if requirements change: Are you going to incrementally rewrite your table? Are you going to export it and bulk load a fixed data model? How much is that going to cost?

Re: The DynamoDB Paper

#77

Post by 0xbadcafebee »

mabbo wrote: post #21

Rick Houlihan did a talk a few years ago about designing the data later for an application using dynamodb. The most common reaction I get from people I show it to- most of them Amazon SDEs who operate services that use Dynamodb- is "Holy shit what is this wizardry?!" https://youtu.be/HaEPXoXVf2k One of the biggest mistakes people make with dynamo is thinking that it's just a relational database with no relations. It'…

Can be performant, nowadays anyway. Worked with a team who built their own implementation because Amazon's was too slow and expensive.

It's a weird model. Too small of a dataset and it doesn't quite make sense to use Dynamo. Too big of a dataset and it's full of footguns. Medium-sized may be too expensive.

Re: The DynamoDB Paper

#79

Post by GauntletWizard »

Twirrim wrote: post #75
davidjfelix wrote: post #46

Earlier quoted context omitted.

It's worth noting that a lot of the early database designs, including this 2018 video pre-date some dramatic improvements to dynamodb usability. I think the biggest ones were: - an increase in the number of GSIs you can create (Dec 2018) [1] - making on-demand possible [2] - an increase in the default limit for number of tables you can create (Mar 2022) [3] I don't think these new features necessarily make the single…

Something else important to mention is that dynamodb now re-consolidates tables. This is a lousy explanation, but Read/Write quota is split evenly over all partitions. Each partition is created based on the hash-key used, and there's an upper limit on how much data can be stored in any given partition. So if you end up with a hot hash-key, lots of stuff in it, that data gets split over more and more and more partitio…

This bit me badly - An application that did significant autoscaling, and hit a peak of 30,000 read/write requests per second - But typically did more like 300.

The conversation with the Amazon support engineer told us that we had over a hundred partitions (which even he admitted was high for that number), and so our quota was effectively giving us 0 iops per partition. This obviously didn't work, and their only solution was "scale it back up, copy everything to a new table". Which we did, but was an engineering effort I'd rather have avoided.

Re: The DynamoDB Paper

#80

Post by josevalerio »

bistablesulphur wrote: post #2

I'be been working with DynamoDB daily for a few years now, and whilst I like working with it and the specific scenario it solves for us, I'd still urge anyone thinking about using it to carefully reconsider whether their problem is truly unique enough that a traditional RDBMS couldn't handle it with some tuning. Theycan be unbelievably performant and give so much stuff for free. Designing application specifically for…

+1

Discovered this while building https://github.com/plutomi/plutomi as I was enamored by Rick's talks and guarantees of `performance at any scale`. In reality, Dynamo was solving scaling issues that we didn't have and the amount of times I've had to rework something to get around some of the quirks of Dynamo led to a lot of lost dev time.

Now that the project is getting more complex, doing simple things such as "searching" (for our use case) are virtually impossible without hosting an ElasticSearch cluster where a simple like %email% in postgres would have sufficed.

Not saying it's a bad DB at all, but you really need to know your access patterns and plan accordingly. Dynamo streams are a godsend and combined with EventBridge you can do some powerful things for asynchronous events. Not paying while it's not running with on demand is awesome, and the performance is truly off the charts. Just please know what you are getting into. In fact, I'd recommend only using Dynamo if you are migrating a "finished" app vs using it for apps that are still evolving

Post reply on HN