Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

There are very few suitable use cases for DynamoDB

syslog.ravelin.com

Post reply on HN
Print view View original

81–90 of 146 posts

Re: There are very few suitable use cases for DynamoDB

#81

Post by leggomylibro »

brandur wrote: post #8

I've seen two large usages of DynamoDB at two different companies, and for what it's worth, in both cases we've had similar trouble as the author. In one case we ended up ripping it out and moving to a sharded Postgres scheme, and in the other we've left in place for now because a migration will be such a monumental effort, but it's pretty much universally maligned. Fundamentally, the problem seems to be that choosin…

Hot partitions do always seem to be an issue, for an 'infinitely scalable' NoSQL database that is a bit on the pricey side.

But what's a better option for a distributed, managed database-as-a-service? Rolling your own does mean significant operational burden.

I wonder if you could do something like add a random number to each of your keys before hashing. It would increase your storage size by Nx, but it seems like that would spread your load as well.

Re: There are very few suitable use cases for DynamoDB

#82

Post by vosper »

leggomylibro wrote: post #81
brandur wrote: post #8

I've seen two large usages of DynamoDB at two different companies, and for what it's worth, in both cases we've had similar trouble as the author. In one case we ended up ripping it out and moving to a sharded Postgres scheme, and in the other we've left in place for now because a migration will be such a monumental effort, but it's pretty much universally maligned. Fundamentally, the problem seems to be that choosin…

Hot partitions do always seem to be an issue, for an 'infinitely scalable' NoSQL database that is a bit on the pricey side. But what's a better option for a distributed, managed database-as-a-service? Rolling your own does mean significant operational burden. I wonder if you could do something like add a random number to each of your keys before hashing. It would increase your storage size by Nx, but it seems like th…

TFA suggests Google's BigTable

Re: There are very few suitable use cases for DynamoDB

#83

Post by manojlds »

lwansbrough wrote: post #26
windlep wrote: post #12

I don't know what the article author is storing, but it's noted that 10GB of data is stored per node. That's quite a bit of data for a single table, and the 10GB is per shard of a table, not per 'database' (DynamoDB only has a notion of tables). Amazon has deep dive talks on DynamoDB on YouTube[1] that go into lots of these details and how to avoid problems from them. It's not that different from understanding how to…

We store 100-1000 GB of data in each wide table on our untrendy on-prem SQL server boxes. 10GB is peanuts. In fact, I propose (given the limitations suggested by the author) that DynamoDB may have NO practical uses worth exploring. If NoSQL is about scale, and it can't scale, what's it good for? I can understand having to optimize your key space, but in this case it necessitates extreme premature optimization.

From my understanding, node is virtual nodes. This article on Cassandra should give some info - https://docs.datastax.com/en/cassandra/2.1/cassandra/archite...

Re: There are very few suitable use cases for DynamoDB

#84

Post by manojlds »

throwaway91111 wrote: post #65
windlep wrote: post #30

Earlier quoted context omitted.

I said it was per node (which is a shard), it's not a limit per table. Did you read the original article or my reply? Both of them state this is per node and how DynamoDB decides when to shard to more nodes. Edit: Sorry, I did mention 10GB originally in relation to a table. That was incorrect of course.

I did. However, 10GB still seems extremely small. A commodity postgres, cassandra, or cockroachdb server can serve HUNDREDS of GB per node. Why is the size per node so small for dynamodb? It seems like poor key space design.

This Cassandra article should provide some info on virtual nodes. https://docs.datastax.com/en/cassandra/2.1/cassandra/archite...

Re: There are very few suitable use cases for DynamoDB

#85

Post by zwily »

throwaway91111 wrote: post #65
windlep wrote: post #30

Earlier quoted context omitted.

I said it was per node (which is a shard), it's not a limit per table. Did you read the original article or my reply? Both of them state this is per node and how DynamoDB decides when to shard to more nodes. Edit: Sorry, I did mention 10GB originally in relation to a table. That was incorrect of course.

I did. However, 10GB still seems extremely small. A commodity postgres, cassandra, or cockroachdb server can serve HUNDREDS of GB per node. Why is the size per node so small for dynamodb? It seems like poor key space design.

10GB is per partition, not per node.

Re: There are very few suitable use cases for DynamoDB

#86

Post by jjirsa »

ZGF4 wrote: post #79

There are a few misguided views in this article and in some of these comments. 1. Every shardable database (Cassandra, Dynamo, BigTable) has to worry about hot spots. Picking a UUID as a partition key is only step one. What happens if one user is a huge majority of your traffic? All of their reads/writes are going to a single partition and of course you are going to suffer from performance issues from that hot spot.…

If one user is the majority of your traffic, you don't shard on user ID.

Re: There are very few suitable use cases for DynamoDB

#87

Post by jjirsa »

TheAceOfHearts wrote: post #53

Many of the comments here are saying that the author's use-case wasn't a good one for DynamoDB. Can anyone share some simple approachable resources that talk about the kinds of use-cases where these tools make sense? Whenever I read about NoSQL systems, I'm always left a unsure about its use-cases. I've only worked on systems where a traditional RDBMS made the most sense. How do you identify when it's appropriate to…

Can't talk about current employer, but precious employer ran thousands of Cassandra nodes to capture and index billions of events a day into a real time graph. Talking petabytes of data, millions of writes per second, in order to fight hackers.

Doing that with an RDBMS would be ... unpleasant. Doing it with Cassandra isn't trivial, but it's straightforward, cross-cloud capable, cross-DC HA, linear scaling, and tunable consistency.

Data integrity is a nonissue - when you're dealing with the scale nosql was meant to solve, you're probably smart enough to learn how to write multiple places safely.

Re: There are very few suitable use cases for DynamoDB

#88

Post by jjirsa »

The best argument against dynamodb is the aws "well architected" guidelines - how are you designing for resiliency with your single region, active-passive database with clunky bolt on replication using kinesis?

Cheaper active-active-active options exist that don't require manual dr failover drills and manual failback when regions inevitably crash.

Re: There are very few suitable use cases for DynamoDB

#89

Post by wora »

windlep wrote: post #76
mbrukman wrote: post #61

Earlier quoted context omitted.

Disclaimer: I am the PM of Google Cloud Bigtable. It appears you're confusing Bigtable with Datastore (to be fair, Datastore is built on Megastore, which is built on Bigtable, so it's an understandable confusion), but let's be clear: Google Cloud Bigtable != Google Cloud Datastore. The URL you cited about consistency models: https://cloud.google.com/datastore/docs/articles/balancing-s... is entirely about Datastore,…

Sorry about that, I had difficulty finding anything about the consistency model of BigTable. Upon further reading on SSTables and how BigTable distributes them across nodes, it appears there is full consistency but if a node goes down the data on it is inaccessible?

If a node goes down, another node will replace it quickly. The node (aka tablet server) doesn't own any data. The data is stored on lower level storage layer.

Re: There are very few suitable use cases for DynamoDB

#90

Post by x0x0 »

qaq wrote: post #54

I can see the future: article that outlines gotchas of Bigtable article you probably shouldn’t use Bigtable article the amount of money we could've saved using PG and not rewriting things 3 times. anything up to 10-15TB there are very few reasons not to use something like PG

And pg supports schema-free data (that they can index!) as columns in your standard sql tables. Check out jsonb if you haven't tried it. It's really really good.

https://www.postgresql.org/docs/9.6/static/datatype-json.htm...

Post reply on HN