Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

There are very few suitable use cases for DynamoDB

syslog.ravelin.com

Post reply on HN
Print view View original

121–130 of 146 posts

Re: There are very few suitable use cases for DynamoDB

#121

Post by qaq »

phamilton wrote: post #119
qaq wrote: post #102

Earlier quoted context omitted.

You can get a box with 48TB RAM so 20TB has more to do with time it takes for DB to come up. To provision DynamoDB to the same performance as PG on i3.16xlarge instances you will be paying north of 50K/month vs 5K/month and that's with all limitations that come with DynamoDB.

An i3.16xlarge is 488GB, not 48TB. The x1.32xlarge (largest amazon offers) is 2TB. Additionally, self-managed PG is a huge operational undertaking compared to DynamoDB. RDS is a bit closer, but the largest RDS offers is db.r3.8xlarge, which is 244GB. The db.r3.8xlarge on RDS Postgres all in with 3000 provisioned IOPS (just provisioning for writes, all reads should be served from memory) and 1.5TB storage ends up arou…

Is there any place in the post I claimed i3.16xlarge has 48TB RAM? I said you can get a box with 48TB RAM. Did anywhere in the post I compared to RDS? i3.16xlarge is basically a dedicated box you have 8 NVMe PCI SSDs to be on super conservative side I was using 50,000 write IOPS 200K read IOPS in reality it can do way more.

Re: There are very few suitable use cases for DynamoDB

#122

Post by datashovel »

Generally speaking. Given that DynamoDB is a NoSQL database service, I'm not certain that moving larger clients to their own dedicated AWS resources should cause too many negative side effects. Especially ones who are so large they're causing scalability issues.

Re: There are very few suitable use cases for DynamoDB

#123

Post by jjirsa »

sametmax wrote: post #91
ZGF4 wrote: post #79

There are a few misguided views in this article and in some of these comments. 1. Every shardable database (Cassandra, Dynamo, BigTable) has to worry about hot spots. Picking a UUID as a partition key is only step one. What happens if one user is a huge majority of your traffic? All of their reads/writes are going to a single partition and of course you are going to suffer from performance issues from that hot spot.…

It depends a lot of the write. If they can be batched, you can put them in a queue or in redis until it reach a threshold and write the update down in the RDBMS. It won't work for all the use cases, but more often that people think.

If you have a persistent data problem, and introduce redis, then you'll have two persistent data problems.

Re: There are very few suitable use cases for DynamoDB

#124

Post by jjirsa »

pishpash wrote: post #108
dserban wrote: post #101

My background is in Cassandra and one company I worked for last year insisted that we use DynamoDB for a project. Here are a few things that ended up being show stoppers. 1. Both the partition key and the sort key are capped at 1 field. In an attempt to "think Cassandra data model", the ugly workaround was to stringify and concatenate things at the application layer, then parse / split on the other side. This made th…

Cassandra isn't any better in those regards. For 1, Cassandra just does the concatenation under the hood and 3 happens all the time on Cassandra. The broken thing is using NoSQL as a DB, not Cassandra or DynamoDB.

Cassandra allows multiple clustering columns, it's not simple concatenation under the hood.

There are no hidden exponential backoffs in Cassandra in any of the hot query paths, period.

Suggesting that using nosql as a DB is broken is so ludicrous I'm not even going to try to refute it - you simply don't know what you're talking about.

Re: There are very few suitable use cases for DynamoDB

#125

Post by nw-fcx »

xendo wrote: post #94
leggomylibro wrote: post #81

Earlier quoted context omitted.

Hot partitions do always seem to be an issue, for an 'infinitely scalable' NoSQL database that is a bit on the pricey side. But what's a better option for a distributed, managed database-as-a-service? Rolling your own does mean significant operational burden. I wonder if you could do something like add a random number to each of your keys before hashing. It would increase your storage size by Nx, but it seems like th…

Keys are the main way to access data in Dynamo. They need to be deterministic to provide fast lookup.

I've been able to improve my partitioning by adding a random suffix from a small set to the key for the writes, and then just trying all suffixes for the reads. Because my work load was relatively write-heavy (used at least ~10x more write capacity than read capacity), and because most of the read capacity was consumed by the size of the entries and not the lookups (so I didn't even have a linear increase in the need for read capacity), I think I came out ahead.

Ultimately, we've decided that dynamodb was not really suited to our use case anyway, but I'd say that's mostly because our total data set was fairly tiny and we our writes were not evenly distributed over time, which is just at odds with the capacity model.

Re: There are very few suitable use cases for DynamoDB

#126

Post by nw-fcx »

mnutt wrote: post #10
nikanj wrote: post #7

The gist of this seems to be that DynamoDB becomes a problem if you have millions of customers. Don't worry. You don't. And there will be many good reasons to refactor the architecture before you do.

I often give this same advice, but assuming this is the case, why reach for DynamoDB at all? Are there small-data use cases where DynamoDB makes more sense than RDS or a hosted redis? At least with those, if you do run into scaling issues you haven't locked yourself into dynamo.

For my very small-data use case I would be very happy if I could use DynamoDB as a persistent data store because the operational aspect is like ten lines of Terraform

Re: There are very few suitable use cases for DynamoDB

#127

Post by nw-fcx »

didibus wrote: post #72
didibus wrote: post #68

The article forgets a very important detail: > A single partition can hold approximately 10 GB of data, and can support a maximum of 3,000 read capacity units or 1,000 write capacity units. DynamoDb will also split your data if you provision more then 3000 reads or 1000 writes. And the caveat is that it will not join back the shards if you later reduce the throughput back down. Instead, each shard will just get even…

Another caveat is that when a shard is split because it has reached its max size, the resulting new 2 shards each get half of the throughput the initial shard had, and not a proportion of the total throughput. So say you have 4000 write capacity and 0 reads (for simplicity). DynamoDb will allocate 4 shards for it, each getting 1000 writes. Now say shard 2 gets too big, and goes above 10GB. DynamoDb will split it in t…

I am really surprised by this, I was convinced that the table-wide capacity is re-distributed evenly across shards.

Looks like you're entirely correct: http://docs.aws.amazon.com/amazondynamodb/latest/developergu...

I wonder how many times I scrolled past the explanation without internalizing it.

Re: There are very few suitable use cases for DynamoDB

#128

Post by anarazel »

phamilton wrote: post #119
qaq wrote: post #102

Earlier quoted context omitted.

You can get a box with 48TB RAM so 20TB has more to do with time it takes for DB to come up. To provision DynamoDB to the same performance as PG on i3.16xlarge instances you will be paying north of 50K/month vs 5K/month and that's with all limitations that come with DynamoDB.

An i3.16xlarge is 488GB, not 48TB. The x1.32xlarge (largest amazon offers) is 2TB. Additionally, self-managed PG is a huge operational undertaking compared to DynamoDB. RDS is a bit closer, but the largest RDS offers is db.r3.8xlarge, which is 244GB. The db.r3.8xlarge on RDS Postgres all in with 3000 provisioned IOPS (just provisioning for writes, all reads should be served from memory) and 1.5TB storage ends up arou…

> The db.r3.8xlarge on RDS Postgres all in with 3000 provisioned IOPS (just provisioning for writes, all reads should be served from memory) and 1.5TB storage ends up around $6k. A 3 year reservation will get that down to $3k. > Comparable DynamoDB (3000 write units, 3000 read units, reserved IOPS) comes out to $1500.

Note those aren't really comparable numbers - postgres will often collapse writes from concurrent sessions / statements.

Re: There are very few suitable use cases for DynamoDB

#129

Post by CyberDildonics »

threeseed wrote: post #40
jsemrau wrote: post #2

When I started building my first app in 2011 MongoDB was the rage. So I build the back-end using the futuristic 'No-SQL' technology. It turned out to be slow (~1 min Query time), inconsistent, and missing an RDBMS layer. Move the thing to PHP/Mysql problems were gone. I still have not found a use case outside of web (comments/discussion) sites where the high integration with Javascript actually makes sense.

Why did you use MongoDB if your domain model wasn't suited ? I just don't understand people who complain about a technology and say it is useless for all of these use cases when they couldn't even spend a few hours to do a Spike/POC or some basic data domain design. MongoDB has very clear documentation about what you should or shouldn't use it for. MongoDB is unique in that it is one of the few document stores availa…

What does document store mean and what can that do that other databases can't?

Re: There are very few suitable use cases for DynamoDB

#130

Post by [deleted] »

jjirsa wrote: post #124
pishpash wrote: post #108

Earlier quoted context omitted.

Cassandra isn't any better in those regards. For 1, Cassandra just does the concatenation under the hood and 3 happens all the time on Cassandra. The broken thing is using NoSQL as a DB, not Cassandra or DynamoDB.

Cassandra allows multiple clustering columns, it's not simple concatenation under the hood. There are no hidden exponential backoffs in Cassandra in any of the hot query paths, period. Suggesting that using nosql as a DB is broken is so ludicrous I'm not even going to try to refute it - you simply don't know what you're talking about.

[deleted]
Post reply on HN