Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

There are very few suitable use cases for DynamoDB

syslog.ravelin.com

Post reply on HN
Print view View original

91–100 of 146 posts

Re: There are very few suitable use cases for DynamoDB

#91

Post by sametmax »

ZGF4 wrote: post #79

There are a few misguided views in this article and in some of these comments. 1. Every shardable database (Cassandra, Dynamo, BigTable) has to worry about hot spots. Picking a UUID as a partition key is only step one. What happens if one user is a huge majority of your traffic? All of their reads/writes are going to a single partition and of course you are going to suffer from performance issues from that hot spot.…

It depends a lot of the write. If they can be batched, you can put them in a queue or in redis until it reach a threshold and write the update down in the RDBMS. It won't work for all the use cases, but more often that people think.

Re: There are very few suitable use cases for DynamoDB

#92

Post by foota »

brandur wrote: post #8

I've seen two large usages of DynamoDB at two different companies, and for what it's worth, in both cases we've had similar trouble as the author. In one case we ended up ripping it out and moving to a sharded Postgres scheme, and in the other we've left in place for now because a migration will be such a monumental effort, but it's pretty much universally maligned. Fundamentally, the problem seems to be that choosin…

If you have significantly more buckets than users though shouldn't the variations in access pattern even out?

Re: There are very few suitable use cases for DynamoDB

#94

Post by xendo »

leggomylibro wrote: post #81
brandur wrote: post #8

I've seen two large usages of DynamoDB at two different companies, and for what it's worth, in both cases we've had similar trouble as the author. In one case we ended up ripping it out and moving to a sharded Postgres scheme, and in the other we've left in place for now because a migration will be such a monumental effort, but it's pretty much universally maligned. Fundamentally, the problem seems to be that choosin…

Hot partitions do always seem to be an issue, for an 'infinitely scalable' NoSQL database that is a bit on the pricey side. But what's a better option for a distributed, managed database-as-a-service? Rolling your own does mean significant operational burden. I wonder if you could do something like add a random number to each of your keys before hashing. It would increase your storage size by Nx, but it seems like th…

Keys are the main way to access data in Dynamo. They need to be deterministic to provide fast lookup.

Re: There are very few suitable use cases for DynamoDB

#95

Post by xendo »

Jach wrote: post #58
choward wrote: post #31

Earlier quoted context omitted.

How would that help? The same user id will still hash to the same thing.

I think I had the same confusion after reading the GP's last statement. The idea of hashing is that the resulting number will be uniformly distributed across the hash space, as opposed to monotonically increasing bit by bit like an auto-increment or timestamp, but it's not the partition key value itself that needs to be uniformly distributed but your access patterns of the partitions. As the docs note, a device id (a…

Write-through cache solves many of the issues with uneven access patterns.

Re: There are very few suitable use cases for DynamoDB

#96

Post by dijit »

threeseed wrote: post #42
dijit wrote: post #4

Earlier quoted context omitted.

I hope you don't mind if I piggyback on this to echo this sentiment. Although I dislike MySQL for its many gotchas (data corruption level stuff too!) I was looking for a _long_ _long_ time for a high consistency NoSQL database.. we basically need document storage of large binary data. Ironically literally nothing in NoSQL land does write-through to disk, they just write to vfs and hope it works; additionally, those t…

You are talking complete nonsense. HBase, Cassandra, MongoDB, Riak, Couchbase etc all write through to disk with proper fsync flushes. And I've never heard of any database that has a model where it writes to a virtual file system - whatever that even means. Please provide some specific examples.

HBase, Cassandra, MongoDB, Riak, Couchbase And redis were the main candidates.

Hbase: http://mail-archives.apache.org/mod_mbox/hbase-issues/201307...

Cassandra: (fsync to WAL, not full fsync). https://wiki.apache.org/cassandra/Durability

MongoDB: ... too much wrong here to list, although I hear it's improving in being cluster aware etc.

Redis does support fsync as far as I remember but the write/delete pattern is incredibly sub-optimal, it runs basically out of a WAL by itself and runs very poorly if your dataset does not fit in memory.

Re: There are very few suitable use cases for DynamoDB

#97

Post by amygdyl »

I think the problem is that there are very few suitable length articles that are available for you at no cost, and the debate has been suffering from a lack of understanding in the widest sense of the problem and so little can be found on any website that tries to make a commercial subsistence which is any way better than you scan reading SO randomly the day before you go to present to management for their architectural planning meeting.

/s is I hope self evident

Only seriously for my purposes, the use I get from the typical article on databases which I find is linked to from HN, is a reverse index to the better discussions, long after the discussion is off the front page here.

I'm merely a little confused about the fact that widespread consternation of the quality of geek journalism for programmers is not much more than merely a occasional mention or moan.

Genuinely is a good dose of cynicism in force, which is invisible to me?

I understand that when I see a story about science intended for a general audience, then HN is likely to become home to much greater detail and depth in discussion.

But with a subject line arguing generalised conclusion from a subject matter of database architecture?

Sometimes I think that I'm confused about whether I'm supposed to be confused about the point of the TFA.

Re: There are very few suitable use cases for DynamoDB

#98

Post by nnx »

phamilton wrote: post #77
nikanj wrote: post #7

The gist of this seems to be that DynamoDB becomes a problem if you have millions of customers. Don't worry. You don't. And there will be many good reasons to refactor the architecture before you do.

We have close to 25M monthly active users and DynamoDB works pretty well for our use case. We store user generated content in DynamoDB. Our largest table is over 1 TB. The mapping of users to their generated content is in Postgres. So doing work on behalf of a particular user will generally be distributed across multiple nodes. We've enjoyed the very predictable performance of DynamoDB as well as the operational simp…

How large is user's generated content in average? Have you considered S3 for your use case? Storage in DDB can be quite pricey (when combining storage cost and and read capacity usage cost)

Re: There are very few suitable use cases for DynamoDB

#99

Post by Dunedan »

natekupp wrote: post #47

We use DynamoDB quite a bit at Thumbtack. Our biggest issue is backups - just wrote a short note about our experiences with DynamoDB here: https://medium.com/@natekupp/dynamodb-and-backups-16dba0dbcd...

Oh yes. Backups for DynamoDB are a pain in the ass, especially as AWS doesn't offer an out-of-the-box solution for that.

That Data Pipeline + EMR solution mentioned in the blog post (here is a better link for it: https://docs.aws.amazon.com/datapipeline/latest/DeveloperGui...) has several drawbacks:

- too many moving parts, especially given the track record of EMR

- might not even be available when your requirement is to keep the data in the same AWS region as the DynamoDB table, as only five regions support Data Pipeline

The best approach I've seen so far is to use DynamoDB Streams and an AWS Lambda function to create incremental backups in an versioned S3-bucket. dynamodb-replicator (https://github.com/mapbox/dynamodb-replicator) implements that together with some scripts to do management tasks like back filling an S3 bucket with data which is already in DynamoDB or joining incremental backups into a single file.

It's still pretty unpolished and definitely needs some love, but I think it's the right approach.

Re: There are very few suitable use cases for DynamoDB

#100

Post by lotyrin »

nemothekid wrote: post #75
karmakaze wrote: post #5

TL;DR - Don't use something as coarse as customer_id as a partition key. Alternatively move to GCP/BigTable. Any DynamoDB tuning advice will say how important it is to have well distributed hash keys. As for the second part, why not use Cloud Spanner? I wish AWS had something like it.

>Any DynamoDB tuning advice will say how important it is to have well distributed hash keys. Maybe I'm missing something, but as I understand it he did have well distributed hash keys (I'm assuming his customer ids were random UUIDs). The problem he had however was that he had so much data that a single custom could cause throughput exceptions on a very small amount of ops/s.

Customer UUID would only be well-distributed for queries of customers.

If some customers are more active than others in a given unit time, some customers have more data than others, and customers' data is keyed by customer ID then no, it's not a well distributed key.

Post reply on HN