Rules for Data Modeling with DynamoDB • Hacker News Board

Re: Rules for Data Modeling with DynamoDB

#21

Post by abd12 » Wed, Apr 08, 2020, 5:51 PM UTC

time0ut wrote: post #20

abd12 wrote: post #2

Author here! If you want more, I just released a book on DynamoDB yesterday --> https://www.dynamodbbook.com/ . There's a launch discount for the next few days. The book is highly recommended by folks at AWS, including Rick Houlihan, the leader of the NoSQL Blackbelt Team at AWS[0]. Happy to answer any questions you have! Also available on Twitter and via email (I'm easily findable). [0] - https://twitter.com/houliha…

Rick is awesome. Bought the book without hesitation. Can't wait to dig through it. I've used DynamoDB in production (small scales only sadly) on a number of projects. Definitely done it the wrong way a few times!

We've all been there! :)

Thanks for your support. I really appreciate it. Don't hesitate to hit me up with any questions or feedback.

Re: Rules for Data Modeling with DynamoDB

#22

Post by polymath21 » Wed, Apr 08, 2020, 5:58 PM UTC

Thanks for this Alex! Will definitely check this out. DynamoDBGuide.com was a huge help to me when I was learning serverless last year to build wanderium.com. There's definitely a learning curve for DynamoDB but the performance is really good.

Do you talk about the best way to do aggregations in your book? That's one of the more annoying downsides of DynamoDB that I've kind of had to hack my way around. (I combine DynamoDB streams with a Lambda function to increment/decrement a count)

Re: Rules for Data Modeling with DynamoDB

#23

Post by abd12 » Wed, Apr 08, 2020, 6:01 PM UTC

polymath21 wrote: post #22

Thanks for this Alex! Will definitely check this out. DynamoDBGuide.com was a huge help to me when I was learning serverless last year to build wanderium.com. There's definitely a learning curve for DynamoDB but the performance is really good. Do you talk about the best way to do aggregations in your book? That's one of the more annoying downsides of DynamoDB that I've kind of had to hack my way around. (I combine Dy…

Thank you! Glad you found DynamoDBGuide.com helpful :)

Yep, I do talk about aggregations in the book. One strategy that I've discussed is available in a blog post here[0] and involves using DynamoDB Transactions to handle aggregates.

If you're looking for large-scale aggregates for analytics (e.g. "What are my top-selling items last month?"), I have an Analytics supplement in the Plus package that includes notes on different patterns for analytics w/ DynamoDB.

Let me know if that helps!

[0] - https://www.alexdebrie.com/posts/dynamodb-transactions/#hand...

Re: Rules for Data Modeling with DynamoDB

#24

Post by fulafel » Wed, Apr 08, 2020, 6:05 PM UTC

DynamoDB seems to be so low level that it takes a lot of design and programming effort to get right. Are there any higher level solutions that build on DynamoDB that take care of these things automatically? For example denormalization sounds pretty error prone if you implement it by hand.

Re: Rules for Data Modeling with DynamoDB

#25

Post by jayd16 » Wed, Apr 08, 2020, 6:05 PM UTC

At what point do these auto-sharding databases like DynamoDB become worth the effort these days? You can squeeze a lot out of a single Postgres instance and much more if you go with read replicas or Redis caches.

When you start with a relational model you don't need a priori knowledge of your data access and you get solid performance and guarantees. If you need this access knowledge beforehand, is DynamoDB best for scaling mature products?

Re: Rules for Data Modeling with DynamoDB

#26

Post by [deleted] » Wed, Apr 08, 2020, 6:11 PM UTC

kpmah wrote: post #17

> Normalization was built for a world with very different assumptions. In the data centers of the 1980s, storage was at a premium and compute was relatively cheap. But the times have changed. Storage is cheap as can be, while compute is at a premium. Normalisation isn't primarily about about saving storage, it's about avoiding update anomalies i.e. correctness.

[deleted]

Re: Rules for Data Modeling with DynamoDB

#27

Post by abd12 » Wed, Apr 08, 2020, 6:12 PM UTC

fulafel wrote: post #24

DynamoDB seems to be so low level that it takes a lot of design and programming effort to get right. Are there any higher level solutions that build on DynamoDB that take care of these things automatically? For example denormalization sounds pretty error prone if you implement it by hand.

It's different than a relational database in that you need to model your data to your patterns, rather than model your data and then handle your patterns.

Once you learn the principles, it really is like clockwork. It changes your process, but you implement the same process every time.

Honestly, I think part of the problem is that there's a lot of bad NoSQL content out there. A little standardization of process in this space will go a long way, IMO :)

Re: Rules for Data Modeling with DynamoDB

#28

Post by abd12 » Wed, Apr 08, 2020, 6:15 PM UTC

jayd16 wrote: post #25

At what point do these auto-sharding databases like DynamoDB become worth the effort these days? You can squeeze a lot out of a single Postgres instance and much more if you go with read replicas or Redis caches. When you start with a relational model you don't need a priori knowledge of your data access and you get solid performance and guarantees. If you need this access knowledge beforehand, is DynamoDB best for s…

I just answered this on Twitter, but I think there are two instances where it's a no-brainer to use DynamoDB:

- High-scale situations where you're worried about performance of a relational database, particularly joins, as it scales.

- If you're using serverless compute (e.g. AWS Lambda or AppSync) where traditional databases don't fit well with the connection model.

That said, you can use DynamoDB for almost every OLTP application. It's just more a matter of personal preference as to whether you want to use a relational database or something like DynamoDB. I pick DynamoDB every time b/c I understand how to use it and like the other benefits (billing model, permissions model, performance characteristics), but I won't say you're wrong if you don't choose it in these other situations.

Re: Rules for Data Modeling with DynamoDB

#29

Post by iampims » Wed, Apr 08, 2020, 6:23 PM UTC

jayd16 wrote: post #25

At what point do these auto-sharding databases like DynamoDB become worth the effort these days? You can squeeze a lot out of a single Postgres instance and much more if you go with read replicas or Redis caches. When you start with a relational model you don't need a priori knowledge of your data access and you get solid performance and guarantees. If you need this access knowledge beforehand, is DynamoDB best for s…

What happens when you need to restart that “single Postgres instance” to apply config changes or upgrade to a more powerful instance class? How do you promote a replica to primary without downtime?

Those concerns are mostly gone when you rely on a service like DynamoDB. It's not “free”, it comes with increased complexity at the app level, but it does offer a piece of mind if you can afford the $$$.

Re: Rules for Data Modeling with DynamoDB

#30

Post by nemothekid » Wed, Apr 08, 2020, 6:24 PM UTC

jayd16 wrote: post #25

At what point do these auto-sharding databases like DynamoDB become worth the effort these days? You can squeeze a lot out of a single Postgres instance and much more if you go with read replicas or Redis caches. When you start with a relational model you don't need a priori knowledge of your data access and you get solid performance and guarantees. If you need this access knowledge beforehand, is DynamoDB best for s…

My rule of thumb has become:

If you know all your access pattern and your writes >>> reads, a NoSQL solution will be cheaper to operate than Postgres. Meaning, I believe, for most deployments, you can get the same amount of performance from postgres, but simply at a higher cost (which may be 3-6x at most). Another reason to go with NoSQL is if you are latency sensitive, although I don't think Dynamo falls in this bucket.

NoSQL was also really good for OLAP, but I think now there are several really good OLAP solutions (like Clickhouse for OSS and Redshift/BigQuery in the cloud) that are easier to manage.