The gist of this seems to be that DynamoDB becomes a problem if you have millions of customers. Don't worry. You don't. And there will be many good reasons to refactor the architecture before you do.
We have close to 25M monthly active users and DynamoDB works pretty well for our use case. We store user generated content in DynamoDB. Our largest table is over 1 TB. The mapping of users to their generated content is in Postgres. So doing work on behalf of a particular user will generally be distributed across multiple nodes. We've enjoyed the very predictable performance of DynamoDB as well as the operational simp…
There are very few suitable use cases for DynamoDB
111–120 of 146 posts
Re: There are very few suitable use cases for DynamoDB
#112There are a few misguided views in this article and in some of these comments. 1. Every shardable database (Cassandra, Dynamo, BigTable) has to worry about hot spots. Picking a UUID as a partition key is only step one. What happens if one user is a huge majority of your traffic? All of their reads/writes are going to a single partition and of course you are going to suffer from performance issues from that hot spot.…
Yeah, but the issue with DynamoDB seems to be bursts of access triggering "throughput exceptions" caused a very static bandwidth allocation which is going down with the number of shards and not so graceful handling of overload situations.
It is imho. an anti-pattern to split up the bandwidth like they do. It negates the multiplexing gain for no good reason except a rigid control model.
Re: There are very few suitable use cases for DynamoDB
#113I've seen two large usages of DynamoDB at two different companies, and for what it's worth, in both cases we've had similar trouble as the author. In one case we ended up ripping it out and moving to a sharded Postgres scheme, and in the other we've left in place for now because a migration will be such a monumental effort, but it's pretty much universally maligned. Fundamentally, the problem seems to be that choosin…
If you have significantly more buckets than users though shouldn't the variations in access pattern even out?
Re: There are very few suitable use cases for DynamoDB
#114NoSQL should stay as a quick and dirty solution for storing key-values. When you start to schematize it and use it as a "DB," you are going down the wrong path. That is because NoSQL is a glorified cache, not a DB. It is essentially using the memory on a large number of nodes to buffer bursty throughput, and using background processes to collate the data later onto disk. There is almost no case where an explicit dist…
You're not describing NoSQL. You're describing something like MongoDB specifically. You're right about the specific case, wrong in the general case.
> There is almost no case where an explicit distributed caching or queuing solution backed by a traditional DB isn't strictly better.
Bah.
If you're looking at big enough data sets (think "Walmart scale", not "web scale") then you have to design your data store to cope with this volume. First, it has to be replicated to multiple servers to serve more queries and for reliability, and is an eventually consistent system a consequence (basically any time you use your read replica, or do anything on a multimaster configuration). You'll need something shardable or multimaster to achieve horizontally scalablity and you'll probably also lose your foreign key enforcement and uniqueness constraints when you do this, so you might as well go the distance and fully denormalise everything for consistent access times. (Have you ever had the Postgres query planner suddenly make something take 10x longer than it usually does because your data patterns changed and decided to use a different query plan than it's ever used before? Yeah, that's a world of fun. Now make that thing a time-critical daily batch job and really squirm.)
When you do the things to make this work, you're basically restricting yourself, abandoning database features. As a consequence, you are going to be in for a world of pain, period, sorry. One key advantage of good NoSQL software is that it doesn't give you those features to begin with, so it's harder to get it wrong. It also has data structures designed to make this sort of thing work well and be performant in ways that general purpose databases won't be, on account of being designed for general purposes.
Re: There are very few suitable use cases for DynamoDB
#115Earlier quoted context omitted.
I often give this same advice, but assuming this is the case, why reach for DynamoDB at all? Are there small-data use cases where DynamoDB makes more sense than RDS or a hosted redis? At least with those, if you do run into scaling issues you haven't locked yourself into dynamo.
I would imagine the use of AWS Lambda might be a pretty decent incentive to use DynamoDB. I'm guessing that stateless DynamoDB queries are faster to fire off than dealing with Postgres or MySQL connection setup and teardown.
Re: There are very few suitable use cases for DynamoDB
#116Earlier quoted context omitted.
We have close to 25M monthly active users and DynamoDB works pretty well for our use case. We store user generated content in DynamoDB. Our largest table is over 1 TB. The mapping of users to their generated content is in Postgres. So doing work on behalf of a particular user will generally be distributed across multiple nodes. We've enjoyed the very predictable performance of DynamoDB as well as the operational simp…
Sounds interesting! But if the mapping is stored in Postgres, your system's availability is capped by that.
Re: There are very few suitable use cases for DynamoDB
#117This is somewhat at odds with their top-level messaging which still pushes DynamoDB as the most scalable solution. And perhaps it is... there are some scalability limits to Aurora. Writes are bottlenecked by one instance. 64TB max. I think performance drops when you exceed the in memory cache. But those limits are still quite large.
Basically I sense some tension between the DynamoDB and Aurora teams and I wonder where this is all going to shake out in the long run.
Here's the full quote (I transcribed it so may contain errors):
"The one thing that surprised me is that there are some customers who are moving their nosql workload to aurora. There are two reasons for that. One, it’s a lot easier to use Aurora because it’s mysql compatible compared to nosql because the interfaces and transaction characteristics are so different. What is also interesting is people also saved money because the IO cost is much lower. In no SQL if you have a large table it gets partitioned then the IO gets partitioned across all the table partitions that you have. And if you have one partition that is hot then you have to provision based on the IO requirement of the hot partition. In the case of Aurora we do automatic heat management so we don’t have this hot parition issue. Second we don’t charge based on provisioned IO. It’s only the IO that you use. And that actually saves a lot of money. In this particular case this is a big social company, interaction company, I cannot tell the name, and they reduced their operational costs by 40% by moving from nosql to Aurora" [1]
Re: There are very few suitable use cases for DynamoDB
#118When I started building my first app in 2011 MongoDB was the rage. So I build the back-end using the futuristic 'No-SQL' technology. It turned out to be slow (~1 min Query time), inconsistent, and missing an RDBMS layer. Move the thing to PHP/Mysql problems were gone. I still have not found a use case outside of web (comments/discussion) sites where the high integration with Javascript actually makes sense.
Why did you use MongoDB if your domain model wasn't suited ? I just don't understand people who complain about a technology and say it is useless for all of these use cases when they couldn't even spend a few hours to do a Spike/POC or some basic data domain design. MongoDB has very clear documentation about what you should or shouldn't use it for. MongoDB is unique in that it is one of the few document stores availa…
Because MongoDB's marketing sold it as the hot new datastore that made SQL legacy - "newer! faster! web scale!" not "only use this if your data isn't relational".
The reference documentation might be more accurate but the way it was publicised certainly wasn't.
Re: There are very few suitable use cases for DynamoDB
#119Earlier quoted context omitted.
20TB in an RDBMS has very different characteristics than 100GB in an RDBMS. Once you can no longer store your dataset in memory, you start to see greater variance in latency. DynamoDB for us has given us very reliable latency for our datasets that are too large to hold in memory on an RDBMS.
You can get a box with 48TB RAM so 20TB has more to do with time it takes for DB to come up. To provision DynamoDB to the same performance as PG on i3.16xlarge instances you will be paying north of 50K/month vs 5K/month and that's with all limitations that come with DynamoDB.
Additionally, self-managed PG is a huge operational undertaking compared to DynamoDB. RDS is a bit closer, but the largest RDS offers is db.r3.8xlarge, which is 244GB.
The db.r3.8xlarge on RDS Postgres all in with 3000 provisioned IOPS (just provisioning for writes, all reads should be served from memory) and 1.5TB storage ends up around $6k. A 3 year reservation will get that down to $3k.
Comparable DynamoDB (3000 write units, 3000 read units, reserved IOPS) comes out to $1500.
The key here is consistent performance and low operational overhead. As I said, we're pretty happy with it.
Re: There are very few suitable use cases for DynamoDB
#120Earlier quoted context omitted.
We have close to 25M monthly active users and DynamoDB works pretty well for our use case. We store user generated content in DynamoDB. Our largest table is over 1 TB. The mapping of users to their generated content is in Postgres. So doing work on behalf of a particular user will generally be distributed across multiple nodes. We've enjoyed the very predictable performance of DynamoDB as well as the operational simp…
How large is user's generated content in average? Have you considered S3 for your use case? Storage in DDB can be quite pricey (when combining storage cost and and read capacity usage cost)
S3 does not do well (cost wise) with billions of files. Provisioning 1000 Write IOPS on Dynamo costs around $100/month. That's 2.5B writes. On S3 that's going to be on the order of $10k. Similarly 1000 Read IOPS on Dynamo is ~$20/month vs 2.5B reads on s3 costing $1000. Storage is cheaper on S3 by about 10x, but per TB that comes out to $250 vs $25, which is hardly a dominating cost factor.