Rules for Data Modeling with DynamoDB • Hacker News Board

Re: Rules for Data Modeling with DynamoDB

#61

Post by davidrupp » Wed, Apr 08, 2020, 8:52 PM UTC

kpmah wrote: post #17

> Normalization was built for a world with very different assumptions. In the data centers of the 1980s, storage was at a premium and compute was relatively cheap. But the times have changed. Storage is cheap as can be, while compute is at a premium. Normalisation isn't primarily about about saving storage, it's about avoiding update anomalies i.e. correctness.

From the abstract of [1], Codd's stated motivation for applying relation theory to data storage:

"Activities of users at terminals and most application programs should remain unaffected when the internal representation of data is changed and even when some aspects of the external representation are changed."

From Section 1.4, "Normal Form":

"A relation whose domains are all simple can be represented in storage by a two-dimensional column-homogeneous array of the kind discussed above. Some more complicated data structure is necessary for a relation with one or more nonsimple domains. For this reason (and others to be cited below) the possibility of eliminating nonsimple domains appears worth investigating. There is, in fact, a very simple elimination procedure, which we shall call normalization."

As I read this, normalization was originally "about" making storage simpler than it was with contemporaneous models. Section 2.3 ("Consistency") discusses how "data bank state" can become inconsistent, and how such inconsistencies might be addressed, up to and including "the user or someone responsible for the security and integrity of the data is notified".

I think it's reasonable to infer that guaranteed transactional consistency (what I think kpmah means above by "correctness") and the space-saving properties of eliminating redundant data both happened later, and both fell out of the initial motivation of simplification.

[1] https://www.seas.upenn.edu/~zives/03f/cis550/codd.pdf -- E. F. Codd, "A Relational Model of Data for Large Shared Data Banks"

Re: Rules for Data Modeling with DynamoDB

#62

Post by [deleted] » Wed, Apr 08, 2020, 9:33 PM UTC

AmericanChopper wrote: post #46

abd12 wrote: post #19

Earlier quoted context omitted.

Great point! It was originally about both of these things, but the storage aspect isn't discussed much anymore because it's really not a concern. The data integrity issue is still a concern, and I talk about that in the book. You need to manage data integrity in your application and think about how to handle updates properly. But it's completely doable and many people have.

> You need to manage data integrity in your application This is just another way of saying you need to implement your own system for managing consistency. Dynamo offers transactions now, but they don’t offer actual serialization. Your transaction will simply fail if it runs into any contention. You might think that’s ok, just retry. But because you’ve chosen to sacrifice modelling your data, this will happen a lot. I…

[deleted]

Re: Rules for Data Modeling with DynamoDB

#63

Post by taffer » Wed, Apr 08, 2020, 9:39 PM UTC

davidrupp wrote: post #61

kpmah wrote: post #17

> Normalization was built for a world with very different assumptions. In the data centers of the 1980s, storage was at a premium and compute was relatively cheap. But the times have changed. Storage is cheap as can be, while compute is at a premium. Normalisation isn't primarily about about saving storage, it's about avoiding update anomalies i.e. correctness.

From the abstract of [1], Codd's stated motivation for applying relation theory to data storage: "Activities of users at terminals and most application programs should remain unaffected when the internal representation of data is changed and even when some aspects of the external representation are changed." From Section 1.4, "Normal Form": "A relation whose domains are all simple can be represented in storage by a t…

Also

[The relational model] provides a basis for a high level data language which will yield maximal independence between programs on the one hand and machine representation and organization of data on the other.

A further advantage of the relational view is that it forms a sound basis for treating derivability, redundancy, and consistency of relations [...]

So I would say it is mainly about flexibility, correctness and the possibility to create a simple yet powerful query language.

Re: Rules for Data Modeling with DynamoDB

#64

Post by agacera » Wed, Apr 08, 2020, 10:08 PM UTC

dfee wrote: post #59

abd12 wrote: post #28

Earlier quoted context omitted.

I just answered this on Twitter, but I think there are two instances where it's a no-brainer to use DynamoDB: - High-scale situations where you're worried about performance of a relational database, particularly joins, as it scales. - If you're using serverless compute (e.g. AWS Lambda or AppSync) where traditional databases don't fit well with the connection model. That said, you can use DynamoDB for almost every OL…

Does the connection model problem go away when using serverless rds? https://aws.amazon.com/rds/aurora/serverless/

I don't know about aurora serverless.

But aws offers a proxy exactly for this purpose.

https://aws.amazon.com/rds/proxy/

Re: Rules for Data Modeling with DynamoDB

#65

Post by aeyes » Wed, Apr 08, 2020, 10:14 PM UTC

ChicagoDave wrote: post #36

There’s an aspect to software development relating to speed/agility. NoSQL data stores offer a schemaless approach That reduces a great deal of unnecessary friction. The amount of time we’ve spent managing relational schemas is ludicrously expensive. There are still great usage patterns for relational, but operationally is not one of them. I’d argue it’s an anti-pattern.

I would not call NoSQL data stores more agile. While it is true that you can add new columns at any point in time the same cannot be said about supporting new access patterns you didn't design for.

Re: Rules for Data Modeling with DynamoDB

#66

Post by __afk__ » Wed, Apr 08, 2020, 10:43 PM UTC

mabbo wrote: post #57

kpmah wrote: post #17

> Normalization was built for a world with very different assumptions. In the data centers of the 1980s, storage was at a premium and compute was relatively cheap. But the times have changed. Storage is cheap as can be, while compute is at a premium. Normalisation isn't primarily about about saving storage, it's about avoiding update anomalies i.e. correctness.

I'm reminded of being 18, my first year of college, and I had this lovely database professor who was good with crowds. He'd get us all to memorize and repeat, as a group certain phrases. "Why do we normalize?" 150 students, in unison: "To make better relations" "And why do we DE-normalize?" 150 students, in unison: "Performance" "And what is a database?" 150 students, in unison: "A self-describing set of integrated r…

Self-describing?

Maybe to someone who could make sense of the DDL and read the language the label col names are written in. And understand all the implicit units, rules around nulls/empties, and presence of magic strings (SSN, SKU) and special numbers (-1) and on and on. For that you need something like RDF and a proper data model.

Re: Rules for Data Modeling with DynamoDB

#67

Post by arborix » Wed, Apr 08, 2020, 11:06 PM UTC

aeyes wrote: post #65

ChicagoDave wrote: post #36

There’s an aspect to software development relating to speed/agility. NoSQL data stores offer a schemaless approach That reduces a great deal of unnecessary friction. The amount of time we’ve spent managing relational schemas is ludicrously expensive. There are still great usage patterns for relational, but operationally is not one of them. I’d argue it’s an anti-pattern.

I would not call NoSQL data stores more agile. While it is true that you can add new columns at any point in time the same cannot be said about supporting new access patterns you didn't design for.

People often confuse startup speed with agility. The talk "Agility ≠ Speed" does a good job of critiquing that idea. Agility is about the ability to change direction gracefully. NoSQL definitely gets you started quickly—as OP noted, you avoid much of the overhead of data modeling and making changes to schemas—but at a certain point you can get into real trouble, and that will slam the brakes on your velocity. I've experienced the same thing with weakly typed languages. These sorts of tools can be very useful in the right use cases, but trading away some of the guardrails can extract a large cost if you aren't thoughtful and careful.

Re: Rules for Data Modeling with DynamoDB

#68

Post by dfee » Wed, Apr 08, 2020, 11:33 PM UTC

agacera wrote: post #64

dfee wrote: post #59

Earlier quoted context omitted.

Does the connection model problem go away when using serverless rds? https://aws.amazon.com/rds/aurora/serverless/

I don't know about aurora serverless. But aws offers a proxy exactly for this purpose. https://aws.amazon.com/rds/proxy/

Very interesting. Thank you for sharing.

Re: Rules for Data Modeling with DynamoDB

#69

Post by WatchDog » Wed, Apr 08, 2020, 11:50 PM UTC

drchopchop wrote: post #37

jayd16 wrote: post #25

At what point do these auto-sharding databases like DynamoDB become worth the effort these days? You can squeeze a lot out of a single Postgres instance and much more if you go with read replicas or Redis caches. When you start with a relational model you don't need a priori knowledge of your data access and you get solid performance and guarantees. If you need this access knowledge beforehand, is DynamoDB best for s…

Dynamo is sort of in-between Redis and SQL: - Less maintenance around schema/migrations - Data types and validation - You still get queries (though not to the level of SQL complexity) - You still get indexes - You get row-level TTL's like Redis - Hosted / infinite scale - Billing based on storage/throughput, not fixed instance sizes

> Less maintenance around schema/migrations

I would say there is much more maintenance, around schema and migrations. Since there is no enforcement of schema at the database level, you need to be very careful in understanding every single way your application(s) work with the data, and ensure that they are backwards and forward compatible. This generally involves writing a lot of custom tooling batch migration logic and ensuring strict control over code that modifies data.

It's very easy to discover schema migration problems in production as the data is accessed.

Re: Rules for Data Modeling with DynamoDB

#70

Post by reilly3000 » Thu, Apr 09, 2020, 1:26 AM UTC

This is a great post, and DDB is a great database for the right use cases. I want to give a shout out to FaunaDb to anybody looking for alternatives - its also serverless and crazy scalable, and usage-based pricing. Its downside is its proprietary FQL query language, not because it sucks (it doesn't!) but there is a learning curve. They provide a rich GraphQL interface as an alternative to FQL. Its upside vs DDB is a much richer set of functions including aggregations, and first-class support for user defined functions. Their attribute-based permissions system is phenomenal. Its definitely worth a look if you're considering DynamoDb but want something that takes less upfront planning about access patterns.