Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Rules for Data Modeling with DynamoDB

trek10.com

Post reply on HN
Print view View original

31–40 of 84 posts

Re: Rules for Data Modeling with DynamoDB

#31

Post by meowface »

iampims wrote: post #29
jayd16 wrote: post #25

At what point do these auto-sharding databases like DynamoDB become worth the effort these days? You can squeeze a lot out of a single Postgres instance and much more if you go with read replicas or Redis caches. When you start with a relational model you don't need a priori knowledge of your data access and you get solid performance and guarantees. If you need this access knowledge beforehand, is DynamoDB best for s…

What happens when you need to restart that “single Postgres instance” to apply config changes or upgrade to a more powerful instance class? How do you promote a replica to primary without downtime? Those concerns are mostly gone when you rely on a service like DynamoDB. It's not “free”, it comes with increased complexity at the app level, but it does offer a piece of mind if you can afford the $$$.

AWS offers a managed Postgres service, too: https://aws.amazon.com/rds/postgresql/

(I'm a fan of DynamoDB and think there are many good use cases for it. Just saying that the above comparison doesn't seem relevant here.)

Re: Rules for Data Modeling with DynamoDB

#32

Post by jugg1es »

Once you get comfortable modeling your data in Dynamo, it becomes hard to justify using RDBMS with all of the overhead that goes along with it. Dynamo is not for every use case, of course, but as long as you design the table generically enough to handle adjacency lists and you don't mind making more than 1 query sometimes, it works really well.

Re: Rules for Data Modeling with DynamoDB

#33

Post by tmpz22 »

meowface wrote: post #31
iampims wrote: post #29

Earlier quoted context omitted.

What happens when you need to restart that “single Postgres instance” to apply config changes or upgrade to a more powerful instance class? How do you promote a replica to primary without downtime? Those concerns are mostly gone when you rely on a service like DynamoDB. It's not “free”, it comes with increased complexity at the app level, but it does offer a piece of mind if you can afford the $$$.

AWS offers a managed Postgres service, too: https://aws.amazon.com/rds/postgresql/ (I'm a fan of DynamoDB and think there are many good use cases for it. Just saying that the above comparison doesn't seem relevant here.)

Managed postgres services tend to be fairly expensive for production usecases at any small - medium organization, and they all come with their own little caveats here and there.

Re: Rules for Data Modeling with DynamoDB

#34

Post by morcutt »

jugg1es wrote: post #32

Once you get comfortable modeling your data in Dynamo, it becomes hard to justify using RDBMS with all of the overhead that goes along with it. Dynamo is not for every use case, of course, but as long as you design the table generically enough to handle adjacency lists and you don't mind making more than 1 query sometimes, it works really well.

I disagree. I spent a few months with it and it was a bit of a nightmare. It fits certain use cases and AWS and others push it as a good use case for more than it should be.

Have geo data? Need a count? Need to look something up for your team member and you don't have time to setup BI tools? Want to query something differently because of design changes? The answer is pretty much no or you can but it'll take some extra thing for all of those.

We ended up picking up an Aurora Serverless and it is working just fine. We can quickly make changes as business cases change, our entire team can query it for now (until we have fancier tools on the biz end), and we are able to move much more quickly.

Re: Rules for Data Modeling with DynamoDB

#35

Post by spdionis »

meowface wrote: post #31
iampims wrote: post #29

Earlier quoted context omitted.

What happens when you need to restart that “single Postgres instance” to apply config changes or upgrade to a more powerful instance class? How do you promote a replica to primary without downtime? Those concerns are mostly gone when you rely on a service like DynamoDB. It's not “free”, it comes with increased complexity at the app level, but it does offer a piece of mind if you can afford the $$$.

AWS offers a managed Postgres service, too: https://aws.amazon.com/rds/postgresql/ (I'm a fan of DynamoDB and think there are many good use cases for it. Just saying that the above comparison doesn't seem relevant here.)

Managed Postgres still needs to be restarted for config changes and instance resizes for example.

Re: Rules for Data Modeling with DynamoDB

#36

Post by ChicagoDave »

There’s an aspect to software development relating to speed/agility. NoSQL data stores offer a schemaless approach That reduces a great deal of unnecessary friction. The amount of time we’ve spent managing relational schemas is ludicrously expensive. There are still great usage patterns for relational, but operationally is not one of them. I’d argue it’s an anti-pattern.

Re: Rules for Data Modeling with DynamoDB

#37

Post by drchopchop »

jayd16 wrote: post #25

At what point do these auto-sharding databases like DynamoDB become worth the effort these days? You can squeeze a lot out of a single Postgres instance and much more if you go with read replicas or Redis caches. When you start with a relational model you don't need a priori knowledge of your data access and you get solid performance and guarantees. If you need this access knowledge beforehand, is DynamoDB best for s…

Dynamo is sort of in-between Redis and SQL:

- Less maintenance around schema/migrations

- Data types and validation

- You still get queries (though not to the level of SQL complexity)

- You still get indexes

- You get row-level TTL's like Redis

- Hosted / infinite scale

- Billing based on storage/throughput, not fixed instance sizes

Re: Rules for Data Modeling with DynamoDB

#38

Post by jugg1es »

morcutt wrote: post #34
jugg1es wrote: post #32

Once you get comfortable modeling your data in Dynamo, it becomes hard to justify using RDBMS with all of the overhead that goes along with it. Dynamo is not for every use case, of course, but as long as you design the table generically enough to handle adjacency lists and you don't mind making more than 1 query sometimes, it works really well.

I disagree. I spent a few months with it and it was a bit of a nightmare. It fits certain use cases and AWS and others push it as a good use case for more than it should be. Have geo data? Need a count? Need to look something up for your team member and you don't have time to setup BI tools? Want to query something differently because of design changes? The answer is pretty much no or you can but it'll take some extr…

Yep, if you need aggregated data at all for operational purposes, Dynamo isn't a good choice. A lot of the hardship people have with Dynamo is not really understanding how to model their data correctly. If you don't design it correctly from the start, you are f*cked, and I sympathize with that.

Re: Rules for Data Modeling with DynamoDB

#39

Post by thoraway1010 »

I love postgresql.

Did business with a startup, signed up, started getting service, they play an intermediary biller / payor role.

Because of an issue in company name used in signup their billing system fell over and didn't setup billing.

But what was crazy is a quickly realized this shop was a noSQL shop. NOTHING connected to anything - so since they hadn't built any reports to cross check any of this they literally did not notice (I noticed other consistency issues elsewhere).

In a SQL database this stuff especially around accounting / money is 101 stuff, but noSQL seemed to really struggle here based on how they'd set it up.

I finally bugged them to charge us, but even that was a bit goofy (basically it looked they exported some transaction to a credit card system - but I doubt had any logic to handle failed payment issues etc).

We have one other vendor where the amount actually charged is a few pennies off the receipt totals - issues with doubles, rounding and application logic or something which doesn't simply use the same number (from database) for order / item detail and total and billing.

So at least in finance / accounting, a ledger which is basically a source of truth, and is linked / summarized / etc in various ways to other parts of systems (sales tax, receipts, credit card charges by order etc) really results in some free consistency wins that don't seem to free in nosql land.

Re: Rules for Data Modeling with DynamoDB

#40

Post by debrice »

There is a lot more you should learn about DynamoDB but I appreciate the effort of the author. Please read the AWS documentation, it's not that big and explains vital things that just aren't in this article. Very important things like:

- LSI can't be created after table is created

- GSI late creation generate back pressure on main table

- If you have an LSI, your table will not scale beyond 10GB

- How often a table will scale up and down per day/hour?

- Cost of auto-scaling in cloudwatch (alarms aren't free)

...and so much more. I've been working with Dynamodb for over 2 years now and I love it.

Post reply on HN