Live data from Hacker News

The DynamoDB Book: Data Modeling with NoSQL and DynamoDB

dynamodbbook.com

21–30 of 114 posts

Re: The DynamoDB Book: Data Modeling with NoSQL and DynamoDB

#21
DynamoDB is very compelling for performance, scalability, and low ops overhead, but I recommend thinking very carefully about the limited transaction support before going with it, as it’s likely to be a dealbreaker for many use cases, whether or not you realize that up front. I think most apps will need a transaction involving more than 25 rows at some point, and with dynamo your only option is to fire them off in groups of 25 and hope none fail (plenty will at scale).

You can get many of the benefits of dynamo (sans auto-sharding), by applying its elegant indexing strategy to an sql database. It will be as fast or faster, your transactions can be as big as you need them to be, and you retain the ability to occasionally fire off un-indexed ad hoc queries for development or convenience. Running and scaling an sql db is also fairly painless these days with options like aurora.

Re: The DynamoDB Book: Data Modeling with NoSQL and DynamoDB

#22
post #16
post #3

Waves Author here. Happy to answer any questions folks have about the book, about DynamoDB, or about self-publishing. NoSQL modeling is waaay different than relational modeling. I think a lot of NoSQL advice out there is pretty bad, which results in people dismissing the technology altogether. I've been working with DynamoDB for a few years now, and there's no way I'll go back. The book has been available for about a…

The biggest problem I'm aware of with DynamoDB is the hot key / partition issue[1]. Throughout is distributed evenly across nodes, you can't control how many nodes you have, so you always have a node that's hot either temporarily or permanently and so you end up having to over provision all your nodes to be able to handle that hot case, which ends up costing far more than alternatives. What's your take on this? This…

With instant adaptive capacity, I think quite a few hot key issues are mitigated.

https://aws.amazon.com/blogs/database/how-amazon-dynamodb-ad...

Re: The DynamoDB Book: Data Modeling with NoSQL and DynamoDB

#23
post #20
post #16

Earlier quoted context omitted.

The biggest problem I'm aware of with DynamoDB is the hot key / partition issue[1]. Throughout is distributed evenly across nodes, you can't control how many nodes you have, so you always have a node that's hot either temporarily or permanently and so you end up having to over provision all your nodes to be able to handle that hot case, which ends up costing far more than alternatives. What's your take on this? This…

As of a couple years ago, DynamoDB will redistribute throughput between shards based on usage [1], so in theory this should eliminate the hot shard problem. I haven't had a chance to test this in practice, if anybody has hands-on experience I'd love to hear it. You also finally have a way of identifying hot keys with the terribly named CloudWatch Contributor Insights for DynamoDB. [2] For exceptional use cases, you a…

That sounds like the problem had been solved and my information is just out of date now. Maybe I should give DynamoDB another look now.

Re: The DynamoDB Book: Data Modeling with NoSQL and DynamoDB

#24
I bought the book, I read the book, I've used DynamoDB for awhile. It didn't change my mind. DynamoDB makes tradeoffs in order to run at massive scale, but scale isn't a problem many people need solving when 2TB of RAM fits in a single box. Meanwhile I need to handle eventual consistency, an analytics pipeline, another database for fuzzy search, another geo lookup database, Lambda functions to do aggregations, and a pile of custom code. All while giving up tooling so readily available for the RDBMS world.

In a world where Opex is much higher than Capex DynamoDB might make sense, but for me server costs are 5% of dev costs. And even if it works from a cost perspective, how many AWS services have the console experience ruined by DynamoDB? The UI tricks you into thinking its a data table with sortable columns, but no! DynamoDB limitations strike again and you are off on a journey of endless paging. The cost savings come at the expense of the user.

DynamoDB also isn't fast. 20ms for a query isn't fast, 30ms for an insert isn't fast. Yes its amazingly consistent and faster than other systems holding 500TB, but that isn't a use case for many users.

Re: The DynamoDB Book: Data Modeling with NoSQL and DynamoDB

#25
post #3

Waves Author here. Happy to answer any questions folks have about the book, about DynamoDB, or about self-publishing. NoSQL modeling is waaay different than relational modeling. I think a lot of NoSQL advice out there is pretty bad, which results in people dismissing the technology altogether. I've been working with DynamoDB for a few years now, and there's no way I'll go back. The book has been available for about a…

Bought the book, thank you!

What was your approach to self-publishing here? What tools did you use? If I wanted to publish a book but knew nothing about it, what resources should I read and what approach would you recommend?

Re: The DynamoDB Book: Data Modeling with NoSQL and DynamoDB

#26

DynamoDB is very compelling for performance, scalability, and low ops overhead, but I recommend thinking very carefully about the limited transaction support before going with it, as it’s likely to be a dealbreaker for many use cases, whether or not you realize that up front. I think most apps will need a transaction involving more than 25 rows at some point, and with dynamo your only option is to fire them off in gr…

Agreed. This is a limitation we ran into trying to implement a critical accounting ledger on top of DynamoDB. The transaction model we came up with is formally verified w/ TLA+. We're turning our work into a product: txlayer.com

Re: The DynamoDB Book: Data Modeling with NoSQL and DynamoDB

#27

DynamoDB is very compelling for performance, scalability, and low ops overhead, but I recommend thinking very carefully about the limited transaction support before going with it, as it’s likely to be a dealbreaker for many use cases, whether or not you realize that up front. I think most apps will need a transaction involving more than 25 rows at some point, and with dynamo your only option is to fire them off in gr…

Interesting, idk that I've ever needed a transaction with more than 25 rows.

But I agree in general about the limitations. Having used RDBMSes like Postgres a lot, as well used Cassandra and DynamoDB in production, I would almost certainly not create a new app with DynamoDB as the primary DB. Even if you have an app where you expect to need to scale writes heavily, it's not going to be on all tables equally. For instance, your users table, and related resources that are relatively small and grow linearly with your users, will probably fit fine in a Postgres DB for a very long time. And being able to have normalized models and powerful indexing and querying patterns available is a big benefit.

DynamoDB can work well for a specific sub-system that needs very high scalability. For instance, if you needed to store pairwise info between every user and product combination for some reason. Of if every user can upload a huge number of resources of some type (though the access patterns need to fit dynamodb's constraints, if these are documents or files of some type then another system like S3 or Elasticsearch would probably make more sense). Or if you're tracking advertising views by an advertising identifier or something. Or scraping and importing a bunch of data from other places. In some specific use-cases like this, the downsides vs an RDMS can be very minimal, and the built-in scalability can save you a ton of time vs having to constantly tune and potentially shard your RDBMS system.

But even in these cases, you might have better options depending on your access patterns. For instance if you don't ever need to refer to this data by reading it in an OLTP context, you might want to just write it to a log like Kafka to be ingested into Redshift or HDFS for offline processing or querying.

Re: The DynamoDB Book: Data Modeling with NoSQL and DynamoDB

#28
post #12

DynamoDB is monster scale but... tricky to use and difficult pricing model. The paying for writers / readers thing is strange to me and makes it difficult to scale up for bursts. I recommend not using this tech for most things. You need to know exactly why you want to use it and have a good reason.

> makes it difficult to scale up for bursts Can you tell me why the On Demand mode doesnt work for you?

You need to build exponential-backoff logic into your system to handle waiting for Dynamo to warm up. It doesn't happen instantly.

Re: The DynamoDB Book: Data Modeling with NoSQL and DynamoDB

#29
post #12

DynamoDB is monster scale but... tricky to use and difficult pricing model. The paying for writers / readers thing is strange to me and makes it difficult to scale up for bursts. I recommend not using this tech for most things. You need to know exactly why you want to use it and have a good reason.

> makes it difficult to scale up for bursts Can you tell me why the On Demand mode doesnt work for you?

7x the cost. I find it interesting that the DynamoDB cheer squad points out most databases only run at 10-15% utilisation and are burning money every hour. In the next breath they suggest running on demand "till it hurts" and paying AWS as if they were running at 15% utilisation.

Re: The DynamoDB Book: Data Modeling with NoSQL and DynamoDB

#30

DynamoDB is very compelling for performance, scalability, and low ops overhead, but I recommend thinking very carefully about the limited transaction support before going with it, as it’s likely to be a dealbreaker for many use cases, whether or not you realize that up front. I think most apps will need a transaction involving more than 25 rows at some point, and with dynamo your only option is to fire them off in gr…

Agreed. This is a limitation we ran into trying to implement a critical accounting ledger on top of DynamoDB. The transaction model we came up with is formally verified w/ TLA+. We're turning our work into a product: txlayer.com

> This is a limitation we ran into trying to implement a critical accounting ledger on top of DynamoDB.

Sounds like a perfect use case for a traditional RDBMS. Why Dynamo?

Post reply on HN