There are very few suitable use cases for DynamoDB • Hacker News Board

Re: There are very few suitable use cases for DynamoDB

#11

Post by justinsaccount » Sat, Jul 08, 2017, 12:39 AM UTC

mnutt wrote: post #10

nikanj wrote: post #7

The gist of this seems to be that DynamoDB becomes a problem if you have millions of customers. Don't worry. You don't. And there will be many good reasons to refactor the architecture before you do.

I often give this same advice, but assuming this is the case, why reach for DynamoDB at all? Are there small-data use cases where DynamoDB makes more sense than RDS or a hosted redis? At least with those, if you do run into scaling issues you haven't locked yourself into dynamo.

I'm not sure how much this applies to dynamodb.. I think I couldn't even figure out how to get started with that.

I used simpledb once as part of project. All I needed it for was to store a small amount of state to be shared between some otherwise stateless ec2 instances. So, kind of the perfect use case.. Except I don't think it is made to scale down that small.

I don't know if I provisioned the capacity wrong or what, but it was so flaky. All I needed it to do was a few reads every few minutes.

I think my issue was I had the provisioned scalability set down to the smallest value I could because I only needed to pull down ~10 rows every 60 seconds or so.. But when the app would start and try to sync up, simpledb would throttle it?

I ended up having to just write a cron job to sync the db to a local json file, because doing the reads from inside my main loop would randomly fail or timeout.

Re: There are very few suitable use cases for DynamoDB

#12

Post by windlep » Sat, Jul 08, 2017, 12:45 AM UTC

I don't know what the article author is storing, but it's noted that 10GB of data is stored per node. That's quite a bit of data for a single table, and the 10GB is per shard of a table, not per 'database' (DynamoDB only has a notion of tables).

Amazon has deep dive talks on DynamoDB on YouTube[1] that go into lots of these details and how to avoid problems from them. It's not that different from understanding how to structure data for Cassandra, Mongo, etc. All the NoSQL systems require an understanding of how they work both to structure your data and ensure optimal performance.

For example, maybe one's consistency constraints are better met by DynamoDB instead of BigTable's varying consistency on different query types[2] (the author of this article didn't address consistency at all). With DynamoDB, you can get strongly consistent reads and queries all the time [3].

Overall, it seems like a kind of weak reason to make the strong statement about "probably shouldn't be using DynamoDB". Maybe a better title would be "Understanding drawbacks of large datasets in DynamoDB". I do hope the author understands the consistency changes they may experience in BigTable, as it could easily require large changes to the code-base if strong consistency was assumed on all queries.

[1] https://www.youtube.com/watch?v=bCW3lhsJKfw

[2] https://cloud.google.com/datastore/docs/articles/balancing-s...

[3] http://docs.aws.amazon.com/amazondynamodb/latest/developergu...

Edit: Fixed inconsistency in what the 10GB limit was referring to, not per table, but per node (shard) of the table.

Re: There are very few suitable use cases for DynamoDB

#13

Post by dyeje » Sat, Jul 08, 2017, 12:49 AM UTC

Our team has been using it alongside Postgres as a scalable metric store and things have been pretty good. We had some growing pains tweaking our storage keys and switching to a daily table to avoid partitioning issues, but it's been quite stable for a while now.

Re: There are very few suitable use cases for DynamoDB

#14

Post by tracker1 » Sat, Jul 08, 2017, 12:54 AM UTC

nikanj wrote: post #7

The gist of this seems to be that DynamoDB becomes a problem if you have millions of customers. Don't worry. You don't. And there will be many good reasons to refactor the architecture before you do.

Well, I just spent the past month dealing with issues trying to get around 360M records into Dynamo... At this point, with the number of partitions, I'm unsure if we'll be able to keep reads to a decent level without bumping the throughput to much higher than what's needed.

Re: There are very few suitable use cases for DynamoDB

#15

Post by hox » Sat, Jul 08, 2017, 1:00 AM UTC

The biggest issue faced with DynamoDB from my perspective has been the problem with any hosted service - that is, the operational stability is in the hands of others entrusted with operating a massive multi-tenant system, and any outage cannot follow your own operational recovery mechanisms unless you plan for failover yourself. And the moment you plan for failover, you need to evaluate why you don't just handle the primary system as well.

Re: There are very few suitable use cases for DynamoDB

#16

Post by hox » Sat, Jul 08, 2017, 1:07 AM UTC

brandur wrote: post #8

I've seen two large usages of DynamoDB at two different companies, and for what it's worth, in both cases we've had similar trouble as the author. In one case we ended up ripping it out and moving to a sharded Postgres scheme, and in the other we've left in place for now because a migration will be such a monumental effort, but it's pretty much universally maligned. Fundamentally, the problem seems to be that choosin…

Conpletely agreed; using DynamoDB for primary storage suffers from needing to design data models around the underlying technology. This is true for secondary keys as well, and is even constrained by the odd pricing model that DynamoDB follows.

I've been happy using DynamoDB as a large-scale caching layer, but even that only fits very specific use case criteria.

Re: There are very few suitable use cases for DynamoDB

#17

Post by elvinyung » Sat, Jul 08, 2017, 1:09 AM UTC

hox wrote: post #15

The biggest issue faced with DynamoDB from my perspective has been the problem with any hosted service - that is, the operational stability is in the hands of others entrusted with operating a massive multi-tenant system, and any outage cannot follow your own operational recovery mechanisms unless you plan for failover yourself. And the moment you plan for failover, you need to evaluate why you don't just handle the…

I mean... of course? But I feel like the idea here is that you can "entrust" them with operating the system and not worry about rolling your own failover mechanism, as long as they seem to be upholding their SLAs.

(Of course, cloud vs. bare metal is a long-running debate, but I think I take the position that multitenant clouds are generally cheap enough that it's worth the cost for the extra feature velocity, if you're not truly at scale.)

Re: There are very few suitable use cases for DynamoDB

#18

Post by double-a » Sat, Jul 08, 2017, 1:09 AM UTC

awfullyjohn wrote: post #9

He complains that DynamoDB doesn't work for him, then says you should instead use Google BigTable. But he doesn't offer evidence why you should use BigTable. And just says that it works for him. I don't buy it. I've used BigTable in the past and found it to be infuriating. Now, because it works for him, I'm supposed to believe that BigTable is right for me?

What did you find infuriating about BigTable?

Re: There are very few suitable use cases for DynamoDB

#19

Post by Diederich » Sat, Jul 08, 2017, 1:09 AM UTC

awfullyjohn wrote: post #9

He complains that DynamoDB doesn't work for him, then says you should instead use Google BigTable. But he doesn't offer evidence why you should use BigTable. And just says that it works for him. I don't buy it. I've used BigTable in the past and found it to be infuriating. Now, because it works for him, I'm supposed to believe that BigTable is right for me?

I believe the article specifically notes that BigTable gave far better performance than DynamoDB.

Re: There are very few suitable use cases for DynamoDB

#20

Post by ryandvm » Sat, Jul 08, 2017, 1:16 AM UTC

mnutt wrote: post #10

nikanj wrote: post #7

The gist of this seems to be that DynamoDB becomes a problem if you have millions of customers. Don't worry. You don't. And there will be many good reasons to refactor the architecture before you do.

I often give this same advice, but assuming this is the case, why reach for DynamoDB at all? Are there small-data use cases where DynamoDB makes more sense than RDS or a hosted redis? At least with those, if you do run into scaling issues you haven't locked yourself into dynamo.

I would imagine the use of AWS Lambda might be a pretty decent incentive to use DynamoDB. I'm guessing that stateless DynamoDB queries are faster to fire off than dealing with Postgres or MySQL connection setup and teardown.