Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

There are very few suitable use cases for DynamoDB

syslog.ravelin.com

Post reply on HN
Print view View original

61–70 of 146 posts

Re: There are very few suitable use cases for DynamoDB

#61

Post by mbrukman »

windlep wrote: post #12

I don't know what the article author is storing, but it's noted that 10GB of data is stored per node. That's quite a bit of data for a single table, and the 10GB is per shard of a table, not per 'database' (DynamoDB only has a notion of tables). Amazon has deep dive talks on DynamoDB on YouTube[1] that go into lots of these details and how to avoid problems from them. It's not that different from understanding how to…

Disclaimer: I am the PM of Google Cloud Bigtable.

It appears you're confusing Bigtable with Datastore (to be fair, Datastore is built on Megastore, which is built on Bigtable, so it's an understandable confusion), but let's be clear: Google Cloud Bigtable != Google Cloud Datastore.

The URL you cited about consistency models: https://cloud.google.com/datastore/docs/articles/balancing-s... is entirely about Datastore, but you referred to it as Bigtable.

Re: There are very few suitable use cases for DynamoDB

#62

Post by apetresc »

Something1234 wrote: post #51
TheAceOfHearts wrote: post #44

Earlier quoted context omitted.

I'm not familiarized with DynamoDB, but why would connecting to it be any faster than connecting to Postgres or MySQL?

Perhaps DynamoDB is based around HTTP, rather then requiring socket back and forth actions(uname + pass, auth, query, results vs just a single hit where query + auth details are packed together). Never actually looked at dynamodb, so not sure.

Yes, that is accurate. DB operations to DynamoDB are just AWS API calls, the same as you would use to spin up an EC2 instance or create an S3 bucket.

Re: There are very few suitable use cases for DynamoDB

#63

Post by apetresc »

hox wrote: post #15

The biggest issue faced with DynamoDB from my perspective has been the problem with any hosted service - that is, the operational stability is in the hands of others entrusted with operating a massive multi-tenant system, and any outage cannot follow your own operational recovery mechanisms unless you plan for failover yourself. And the moment you plan for failover, you need to evaluate why you don't just handle the…

There are many things people justifiably complain about DynamoDB, but operational support has rarely been one of them.

Re: There are very few suitable use cases for DynamoDB

#64

Post by imauld »

We DynamoDB to store user alerts and it works fairly well for our use case. However we still run into issues with consumed throughput being well under provisioned but still getting throughput exceptions.

We don't use the auto incremented user ID's either. We create a hash of the user ID and some of the other data contained in the alert and it would appear we still have hot partitions.

Re: There are very few suitable use cases for DynamoDB

#65

Post by throwaway91111 »

windlep wrote: post #30
throwaway91111 wrote: post #27

Earlier quoted context omitted.

It's amazing how much you can put into a reply without directly addressing any of the concerns. Are you serious that dynamodb can't even hold 10GB? If you intended to defend the database I think you failed. Sqlite3 can store 10 gigs without sweating. What on earth can you use dynamo db for if it doesn't scale with the data? The way you describe it seems much WORSE than the OP.

I said it was per node (which is a shard), it's not a limit per table. Did you read the original article or my reply? Both of them state this is per node and how DynamoDB decides when to shard to more nodes. Edit: Sorry, I did mention 10GB originally in relation to a table. That was incorrect of course.

I did. However, 10GB still seems extremely small. A commodity postgres, cassandra, or cockroachdb server can serve HUNDREDS of GB per node. Why is the size per node so small for dynamodb?

It seems like poor key space design.

Re: There are very few suitable use cases for DynamoDB

#66

Post by mbrukman »

awfullyjohn wrote: post #9

He complains that DynamoDB doesn't work for him, then says you should instead use Google BigTable. But he doesn't offer evidence why you should use BigTable. And just says that it works for him. I don't buy it. I've used BigTable in the past and found it to be infuriating. Now, because it works for him, I'm supposed to believe that BigTable is right for me?

Hey there, I'm the PM for Google Cloud Bigtable; if you want to talk about your experiences with Bigtable, I'd love to hear what your use case was, what your experience was, and your thoughts on how we can make Bigtable better for you. Feel free to follow me on Twitter (see my profile) and we can chat over DM if you wish.

If we chat and decide that Bigtable is appropriate for your use case, maybe you'd be willing to give it another shot? :-)

Re: There are very few suitable use cases for DynamoDB

#67

Post by macinjosh »

My company is in the online form builder space and usage of our MySQL RDS instances are nearing the limit of what is offered. I looked into moving certain high-volume/high-traffic data models to DynamoDB and realized we could not because each record is limited to a measly (for our needs) 400kb.

Re: There are very few suitable use cases for DynamoDB

#68

Post by didibus »

The article forgets a very important detail:

> A single partition can hold approximately 10 GB of data, and can support a maximum of 3,000 read capacity units or 1,000 write capacity units.

DynamoDb will also split your data if you provision more then 3000 reads or 1000 writes. And the caveat is that it will not join back the shards if you later reduce the throughput back down. Instead, each shard will just get even less throughput then you might believe.

Re: There are very few suitable use cases for DynamoDB

#69

Post by [deleted] »

mbrukman wrote: post #61
windlep wrote: post #12

I don't know what the article author is storing, but it's noted that 10GB of data is stored per node. That's quite a bit of data for a single table, and the 10GB is per shard of a table, not per 'database' (DynamoDB only has a notion of tables). Amazon has deep dive talks on DynamoDB on YouTube[1] that go into lots of these details and how to avoid problems from them. It's not that different from understanding how to…

Disclaimer: I am the PM of Google Cloud Bigtable. It appears you're confusing Bigtable with Datastore (to be fair, Datastore is built on Megastore, which is built on Bigtable, so it's an understandable confusion), but let's be clear: Google Cloud Bigtable != Google Cloud Datastore. The URL you cited about consistency models: https://cloud.google.com/datastore/docs/articles/balancing-s... is entirely about Datastore,…

[deleted]

Re: There are very few suitable use cases for DynamoDB

#70

Post by windlep »

throwaway91111 wrote: post #65
windlep wrote: post #30

Earlier quoted context omitted.

I said it was per node (which is a shard), it's not a limit per table. Did you read the original article or my reply? Both of them state this is per node and how DynamoDB decides when to shard to more nodes. Edit: Sorry, I did mention 10GB originally in relation to a table. That was incorrect of course.

I did. However, 10GB still seems extremely small. A commodity postgres, cassandra, or cockroachdb server can serve HUNDREDS of GB per node. Why is the size per node so small for dynamodb? It seems like poor key space design.

Agreed entirely, I do wonder what kind of internal constraints led Amazon to have these limits. Maybe in the future they'll go away, as so many other seemingly arbitrary limits do.
Post reply on HN