Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Rules for Data Modeling with DynamoDB

trek10.com

Post reply on HN
Print view View original

51–60 of 84 posts

Re: Rules for Data Modeling with DynamoDB

#51

Post by erik_seaberg »

abd12 wrote: post #10
deckarep wrote: post #8

Earlier quoted context omitted.

How much of the book is directly applicable to Cassandra? I understand that Apache Cassandra is regarded as the “open-source” version of DynamoDB?

The concepts are definitely applicable. You'll need to do some small work to translate vocabulary and work around a slightly different feature set, but most of it should work for you. And yep, Cassandra is pretty similar to DynamoDB. Both are wide-column data stores. Some of the original folks that worked on Dynamo (not DynamoDB) at Amazon.com went to Facebook and worked on Cassandra. The concepts underlying Dynamo b…

If you take Cassandra and remove the power user features whose runtime cost is hard to predict, what you're left with is pretty close to DynamoDB. It's harder to use and incompatible with everything else, but its key feature is not overpromising capacity. We're only considering alternatives because there's no cost saving story around tiered storage.

Re: Rules for Data Modeling with DynamoDB

#52

Post by abd12 »

debrice wrote: post #40

There is a lot more you should learn about DynamoDB but I appreciate the effort of the author. Please read the AWS documentation, it's not that big and explains vital things that just aren't in this article. Very important things like: - LSI can't be created after table is created - GSI late creation generate back pressure on main table - If you have an LSI, your table will not scale beyond 10GB - How often a table w…

One note on this -- if you have an LSI, you can't have an item collection larger than 10GB, where an item collection refers to all the items with the same partition key in your main table and your LSI.

A DynamoDB table with an LSI can scale far beyond 10GB. That said, I would avoid LSIs in almost all circumstances. Just go with a GSI.

Re: Rules for Data Modeling with DynamoDB

#53

Post by k__ »

abd12 wrote: post #2

Author here! If you want more, I just released a book on DynamoDB yesterday --> https://www.dynamodbbook.com/ . There's a launch discount for the next few days. The book is highly recommended by folks at AWS, including Rick Houlihan, the leader of the NoSQL Blackbelt Team at AWS[0]. Happy to answer any questions you have! Also available on Twitter and via email (I'm easily findable). [0] - https://twitter.com/houliha…

Is a HTML version available?

Re: Rules for Data Modeling with DynamoDB

#54

Post by swyx »

mabbo wrote: post #3
abd12 wrote: post #2

Author here! If you want more, I just released a book on DynamoDB yesterday --> https://www.dynamodbbook.com/ . There's a launch discount for the next few days. The book is highly recommended by folks at AWS, including Rick Houlihan, the leader of the NoSQL Blackbelt Team at AWS[0]. Happy to answer any questions you have! Also available on Twitter and via email (I'm easily findable). [0] - https://twitter.com/houliha…

After seeing Rick's Re:invent talk, the one where at about minute 40 everyones' heads exploded, I emailed him (I'm in a very far away other department of Amazon) to ask him for more, because everything he was saying was absolutely not the way my group was using DynamoDB (ie: we were doing it wrong). He could have ignored my email entirely. He's a busy guy, right? I wouldn't have held it against him at all. Instead, h…

aww thats such a nice story to hear :) i would absolutely hope he treats internal customers the same as external.

Re: Rules for Data Modeling with DynamoDB

#55

Post by swyx »

abd12 wrote: post #12
mulmen wrote: post #11

Earlier quoted context omitted.

Very exciting! Is there any plan to release a physical copy? With reference books I much prefer something I can put on my desk.

Thanks! Honestly, I'm right with you. I would love a physical copy. I did a bit of research and didn't find any great options for making a physical copy of a self-published book for the number of copies I'm expecting to sell (given it's a fairly niche technical area). That said, if anyone has any great recommendations here, I'm all ears. Actual experience would be best if possible, rather than the first thing you see…

uh doesnt amazon itself have a physical book publishing service?

Re: Rules for Data Modeling with DynamoDB

#56

Post by swyx »

debrice wrote: post #40

There is a lot more you should learn about DynamoDB but I appreciate the effort of the author. Please read the AWS documentation, it's not that big and explains vital things that just aren't in this article. Very important things like: - LSI can't be created after table is created - GSI late creation generate back pressure on main table - If you have an LSI, your table will not scale beyond 10GB - How often a table w…

sounds like YOU should write a blogpost or five :)

Re: Rules for Data Modeling with DynamoDB

#57

Post by mabbo »

kpmah wrote: post #17

> Normalization was built for a world with very different assumptions. In the data centers of the 1980s, storage was at a premium and compute was relatively cheap. But the times have changed. Storage is cheap as can be, while compute is at a premium. Normalisation isn't primarily about about saving storage, it's about avoiding update anomalies i.e. correctness.

I'm reminded of being 18, my first year of college, and I had this lovely database professor who was good with crowds. He'd get us all to memorize and repeat, as a group certain phrases.

"Why do we normalize?"

150 students, in unison: "To make better relations"

"And why do we DE-normalize?"

150 students, in unison: "Performance"

"And what is a database?"

150 students, in unison: "A self-describing set of integrated records"

That was 16 years ago, and I'm still able to say those answers verbatim.

Re: Rules for Data Modeling with DynamoDB

#58

Post by debrice »

abd12 wrote: post #52
debrice wrote: post #40

There is a lot more you should learn about DynamoDB but I appreciate the effort of the author. Please read the AWS documentation, it's not that big and explains vital things that just aren't in this article. Very important things like: - LSI can't be created after table is created - GSI late creation generate back pressure on main table - If you have an LSI, your table will not scale beyond 10GB - How often a table w…

One note on this -- if you have an LSI, you can't have an item collection larger than 10GB, where an item collection refers to all the items with the same partition key in your main table and your LSI. A DynamoDB table with an LSI can scale far beyond 10GB. That said, I would avoid LSIs in almost all circumstances. Just go with a GSI.

Thank you for the clarification abd12, you are right. I only create LSI if I know for sure the data within a given shard will not go beyond 10GB... since we never really do we always go with GSI.

Re: Rules for Data Modeling with DynamoDB

#59

Post by dfee »

abd12 wrote: post #28
jayd16 wrote: post #25

At what point do these auto-sharding databases like DynamoDB become worth the effort these days? You can squeeze a lot out of a single Postgres instance and much more if you go with read replicas or Redis caches. When you start with a relational model you don't need a priori knowledge of your data access and you get solid performance and guarantees. If you need this access knowledge beforehand, is DynamoDB best for s…

I just answered this on Twitter, but I think there are two instances where it's a no-brainer to use DynamoDB: - High-scale situations where you're worried about performance of a relational database, particularly joins, as it scales. - If you're using serverless compute (e.g. AWS Lambda or AppSync) where traditional databases don't fit well with the connection model. That said, you can use DynamoDB for almost every OL…

Does the connection model problem go away when using serverless rds? https://aws.amazon.com/rds/aurora/serverless/

Re: Rules for Data Modeling with DynamoDB

#60

Post by owenmarshall »

abd12 wrote: post #19
kpmah wrote: post #17

> Normalization was built for a world with very different assumptions. In the data centers of the 1980s, storage was at a premium and compute was relatively cheap. But the times have changed. Storage is cheap as can be, while compute is at a premium. Normalisation isn't primarily about about saving storage, it's about avoiding update anomalies i.e. correctness.

Great point! It was originally about both of these things, but the storage aspect isn't discussed much anymore because it's really not a concern. The data integrity issue is still a concern, and I talk about that in the book. You need to manage data integrity in your application and think about how to handle updates properly. But it's completely doable and many people have.

Are there any basic examples you can give around maintaining that integrity?

I'm liking DynamoDB for tasks that fit nicely within a single domain, have relatively pain-free access patterns, etc. And I've found good fits, but there are some places where the eventual consistency model makes me nervous.

I'm specifically thinking about updating multiple different DynamoDB keys that might need to be aggregated for a data object. The valid answer may be "don't do that!" – if so, what should I do?

(I'll probably just buy the book ;-))

Post reply on HN