Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Rules for Data Modeling with DynamoDB

trek10.com

Post reply on HN
Print view View original

71–80 of 84 posts

Re: Rules for Data Modeling with DynamoDB

#71

Post by tiew9Vii »

abd12 wrote: post #2

Author here! If you want more, I just released a book on DynamoDB yesterday --> https://www.dynamodbbook.com/ . There's a launch discount for the next few days. The book is highly recommended by folks at AWS, including Rick Houlihan, the leader of the NoSQL Blackbelt Team at AWS[0]. Happy to answer any questions you have! Also available on Twitter and via email (I'm easily findable). [0] - https://twitter.com/houliha…

Interested in the book.

It's a bit expensive especially with exchange rates but it is what it is, these things take your time and effort to produce.

From the website I cannot see a table of contents for the book unless this is under provide an email for free chapters. Can you please provide a publicly visible table of contents no email required on the site/here?.

This will be the deciding factor for me making a purchase as I'll be able to see what the book covers and if it'll be of use to me having several years Dynamo experience.

Re: Rules for Data Modeling with DynamoDB

#72

Post by ajxs »

> "With denormalizing, data integrity is more of an application concern. You'll need to consider when this duplicated data can change and how to update it if needed. But this denormalization will give you a greater scale than is possible with other databases."

There's the big catch. As another poster pointed out, normalisation is not about efficiency. It's about correctness. People have been quick to make the comparison between storage and compute cost. The high cost of development and bug-fixing time trumps both of them by an order of magnitude. The guarantee of referential-integrity alone that SQL offers helps eradicate an entire class of bugs for your application with no added effort. This article glosses so blithely over this critical caveat. Whenever this discussion comes up I'm quick to refer back to the yardstick of "Does your application have users? If so, then its data is relational". I can't wait for the day when we look back at NoSQL as the 'dancing sickness' of the IT world.

It's also worth questioning: 'At what scale does this tradeoff become worthwhile?' Another poster here correctly pointed out that modern versions of Postgres scale remarkably well. The tipping point where this kind of NoSQL implementation becomes the most efficient option is likely to be far beyond the scale of most products. It's true that completely denormalising your data will make reads much faster, this is undeniable. This does not mean you need to throw the baby out with the bathwater and store your master data in NoSQL.

Re: Rules for Data Modeling with DynamoDB

#73

Post by femto113 »

> To handle [compound primary keys] in DynamoDB, you would need to create two items in a transaction where each operation asserts that there is not an existing item with the same primary key.

There are other approaches--in cases where I've needed compound keys I've had success using version-5 UUIDs as the primary key constructed from a concatenation of the compound key fields. The advantage is that Dynamo's default optimistic locking works as expected with no transaction needed. A potential disadvantage is if you frequently need to look records up by just one component you'd need a secondary index instead of the primary doing double duty.

Re: Rules for Data Modeling with DynamoDB

#74

Post by AmericanChopper »

owenmarshall wrote: post #60
abd12 wrote: post #19

Earlier quoted context omitted.

Great point! It was originally about both of these things, but the storage aspect isn't discussed much anymore because it's really not a concern. The data integrity issue is still a concern, and I talk about that in the book. You need to manage data integrity in your application and think about how to handle updates properly. But it's completely doable and many people have.

Are there any basic examples you can give around maintaining that integrity? I'm liking DynamoDB for tasks that fit nicely within a single domain, have relatively pain-free access patterns, etc. And I've found good fits, but there are some places where the eventual consistency model makes me nervous. I'm specifically thinking about updating multiple different DynamoDB keys that might need to be aggregated for a data…

> Are there any basic examples you can give around maintaining that integrity?

For those types of use cases, the OP’s advice would actually require implementing a fully bespoke concurrency control system in your business logic layer. Without trying to disparage the OP, this is for all intents and purposes, impossible (aside from also being very, very impractical). There’s some things you can do to create additional almost-functional (though still highly impractical) consistency controls for dynamo (like throttling through FIFO queues), but they all end up being worse performance and scaling trade-offs then you’d get from simply using an RDBMS.

A lot of it boils down to the fact that dynamo doesn’t have (and wasn’t designed to have) locking, meaning that pretty much any concurrency control system you want to implement on top of it, is eventually going to run into a brick wall. The best you’d possibly be able to do is a very, very slow and clunky reimplementation of some of Spanner’s design patterns.

Re: Rules for Data Modeling with DynamoDB

#75

Post by AmericanChopper »

ajxs wrote: post #72

> "With denormalizing, data integrity is more of an application concern. You'll need to consider when this duplicated data can change and how to update it if needed. But this denormalization will give you a greater scale than is possible with other databases." There's the big catch. As another poster pointed out, normalisation is not about efficiency. It's about correctness. People have been quick to make the compari…

> I can't wait for the day when we look back at NoSQL as the 'dancing sickness' of the IT world.

It does have some compelling use cases. It’s just relational data isn’t one of them. If you have a use case with a low potential for write contention, a tolerance for eventual consistency, a very simple data structure, and a high demand for read throughput, then it’s great. One area that I’ve seen it used with great success is content publishing. You have one author, perhaps an additional editor/proofreader, the content is one document (with perhaps one other related document, like an author bio), and hopefully you want thousands or perhaps millions of people to be able to get decent read performance. Another example could be pretty much anything you’d typically use a Materialized View for in a DB. You can compute the view in your RDBMS, periodically publish it to a document database, and then offload all read throughput to a better suited system.

NoSQL is usually used wrong imo, but that doesn’t mean there aren’t ways to use it right. There’s valid use cases for graph databases and stream processing systems too. But they’re not hip enough to produce the same volume of highly questionable web apps.

Re: Rules for Data Modeling with DynamoDB

#76

Post by ajxs »

AmericanChopper wrote: post #75
ajxs wrote: post #72

> "With denormalizing, data integrity is more of an application concern. You'll need to consider when this duplicated data can change and how to update it if needed. But this denormalization will give you a greater scale than is possible with other databases." There's the big catch. As another poster pointed out, normalisation is not about efficiency. It's about correctness. People have been quick to make the compari…

> I can't wait for the day when we look back at NoSQL as the 'dancing sickness' of the IT world. It does have some compelling use cases. It’s just relational data isn’t one of them. If you have a use case with a low potential for write contention, a tolerance for eventual consistency, a very simple data structure, and a high demand for read throughput, then it’s great. One area that I’ve seen it used with great succe…

You're absolutely correct with the point about the materialised view use case. I wasn't going to labor the point going into extra detail in my post. The most successful use I've seen for NoSQL databases is aggregating complex relational data structures into a single document record periodically. You're entirely correct. It's not that the technology is inherently wrong ( in most cases, MongoDB is another story ) it's just the widespread misuse giving this technology a bad name.

Re: Rules for Data Modeling with DynamoDB

#77

Post by abhishektwr »

Having spent a few years working with DynamoDB to build multi-region, multi-tenancy platforms, I must say that DynamoDB is a good fit as a supplement datastore i.e. you should only store a sub-set of information managed by your serverless microservice. DynamoDB multi-region replication is just amazing. Unfortunately, we had a few massive billing spikes with DynamoDB, and we end-up adding pricing measurement and tests to track read/write units in all our functions.

I generally don't recommend DynamoDB as primary data store irrespective of your use case. It takes too much time to model the data. With every new requirement, you have to redo a lot of modelling exercise. Choices you made in beginning start looking bad and you will not remember why you created that particular combination of the composite key or local secondary index which offers no benefit due to incremental changes. Transaction support is painful, existing SDKs just don't cut.

I often wish some of the GCP Firebase features are available in DynamoDB like namespace, control on daily throughput to avoid billing spikes and transaction support.

Re: Rules for Data Modeling with DynamoDB

#78

Post by ChicagoDave »

arborix wrote: post #67
aeyes wrote: post #65

Earlier quoted context omitted.

I would not call NoSQL data stores more agile. While it is true that you can add new columns at any point in time the same cannot be said about supporting new access patterns you didn't design for.

People often confuse startup speed with agility. The talk "Agility ≠ Speed" does a good job of critiquing that idea. Agility is about the ability to change direction gracefully. NoSQL definitely gets you started quickly—as OP noted, you avoid much of the overhead of data modeling and making changes to schemas—but at a certain point you can get into real trouble, and that will slam the brakes on your velocity. I've ex…

If I hadn’t been on large corporate projects that were wildly successful in extreme shortened periods of design and development, I wouldn’t make my arguments.

Adapting to the needs of a given domain may require a relational data store, but it’s my experience those are rare occasions.

Re: Rules for Data Modeling with DynamoDB

#80

Post by ChicagoDave »

There is an unhealthy attachment to relational data stores today. It’s a tool, not an architecture or solution. We shouldn’t start with them and often should exclude them from our operational implementations. Reporting and analysis systems benefit tremendously from relational data stores. But we learned years ago that separate operational and reporting systems provided optimal performance.

I suggest those of you still unfamiliar with nosql operational data storage patterns trust companies like Trek10 and Accenture (where i saw great success).

Post reply on HN