Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

There are very few suitable use cases for DynamoDB

syslog.ravelin.com

Post reply on HN
Print view View original

131–140 of 146 posts

Re: There are very few suitable use cases for DynamoDB

#131

Post by grogenaut »

hox wrote: post #16
brandur wrote: post #8

I've seen two large usages of DynamoDB at two different companies, and for what it's worth, in both cases we've had similar trouble as the author. In one case we ended up ripping it out and moving to a sharded Postgres scheme, and in the other we've left in place for now because a migration will be such a monumental effort, but it's pretty much universally maligned. Fundamentally, the problem seems to be that choosin…

Conpletely agreed; using DynamoDB for primary storage suffers from needing to design data models around the underlying technology. This is true for secondary keys as well, and is even constrained by the odd pricing model that DynamoDB follows. I've been happy using DynamoDB as a large-scale caching layer, but even that only fits very specific use case criteria.

I laugh every time I see this because everyone seems to forget the rough time they had the first time they encountered a relational database and how to map their problem into that space... Sure it's pretty straight forward now but that's the point... You're still doing it... You are still using your domain knowledge of relational to map your problem ilto the underlying tech. You just don't notice it. "Oh companies have people and people have multiple addresses with one marked as primary and addresses have 1-3 address lines and several optional fields based on local and and and and..." That's several years of learning if you had to discover it yourself there, not just on the address domain but also on how to decompose it.

Re: There are very few suitable use cases for DynamoDB

#132

Post by grogenaut »

nw-fcx wrote: post #125
xendo wrote: post #94

Earlier quoted context omitted.

Keys are the main way to access data in Dynamo. They need to be deterministic to provide fast lookup.

I've been able to improve my partitioning by adding a random suffix from a small set to the key for the writes, and then just trying all suffixes for the reads. Because my work load was relatively write-heavy (used at least ~10x more write capacity than read capacity), and because most of the read capacity was consumed by the size of the entries and not the lookups (so I didn't even have a linear increase in the need…

What were you using as keys. I heavily use Dynamo and use uuids for about every key

Re: There are very few suitable use cases for DynamoDB

#133

Post by phamilton »

qaq wrote: post #121
phamilton wrote: post #119

Earlier quoted context omitted.

An i3.16xlarge is 488GB, not 48TB. The x1.32xlarge (largest amazon offers) is 2TB. Additionally, self-managed PG is a huge operational undertaking compared to DynamoDB. RDS is a bit closer, but the largest RDS offers is db.r3.8xlarge, which is 244GB. The db.r3.8xlarge on RDS Postgres all in with 3000 provisioned IOPS (just provisioning for writes, all reads should be served from memory) and 1.5TB storage ends up arou…

Is there any place in the post I claimed i3.16xlarge has 48TB RAM? I said you can get a box with 48TB RAM. Did anywhere in the post I compared to RDS? i3.16xlarge is basically a dedicated box you have 8 NVMe PCI SSDs to be on super conservative side I was using 50,000 write IOPS 200K read IOPS in reality it can do way more.

Ok. Let me try again:

> You can get a box with 48TB RAM

Not with AWS. If you don't have AWS as a requirement, go for it.

> i3.16xlarge

An i3.16xlarge still won't hold a 20TB dataset in memory. You are going to see more variance in latency if you have a working dataset too big to fit into memory.

As far as getting superior performance out of an i3.16xlarge, that's fine if you have the expertise and resources to run PG yourself. However you're going to need replication, which will increase that cost. You're going to need failover mechanisms, backup, etc.

We have preferred RDS for Postgres because it gives us something operationally simple. We've found DynamoDB to be even simpler operationally and have more predictable performance. We have frequently considered running a self managed PG instance and have decided against it for our use case.

Re: There are very few suitable use cases for DynamoDB

#134

Post by qaq »

phamilton wrote: post #133
qaq wrote: post #121

Earlier quoted context omitted.

Is there any place in the post I claimed i3.16xlarge has 48TB RAM? I said you can get a box with 48TB RAM. Did anywhere in the post I compared to RDS? i3.16xlarge is basically a dedicated box you have 8 NVMe PCI SSDs to be on super conservative side I was using 50,000 write IOPS 200K read IOPS in reality it can do way more.

Ok. Let me try again: > You can get a box with 48TB RAM Not with AWS. If you don't have AWS as a requirement, go for it. > i3.16xlarge An i3.16xlarge still won't hold a 20TB dataset in memory. You are going to see more variance in latency if you have a working dataset too big to fit into memory. As far as getting superior performance out of an i3.16xlarge, that's fine if you have the expertise and resources to run PG…

Ok let's try :) Did I ever claim on 48TB on AWS ? You are seriously claiming DynamoDB will be holding full 20TB in RAM if yes I have a bridge to sell you :). The price for i3.16xlarge was quoted for 2 instances the IOPS used for calculation is 1/8th of actual max IOPS that instance can do.

Re: There are very few suitable use cases for DynamoDB

#135

Post by phamilton »

qaq wrote: post #134
phamilton wrote: post #133

Earlier quoted context omitted.

Ok. Let me try again: > You can get a box with 48TB RAM Not with AWS. If you don't have AWS as a requirement, go for it. > i3.16xlarge An i3.16xlarge still won't hold a 20TB dataset in memory. You are going to see more variance in latency if you have a working dataset too big to fit into memory. As far as getting superior performance out of an i3.16xlarge, that's fine if you have the expertise and resources to run PG…

Ok let's try :) Did I ever claim on 48TB on AWS ? You are seriously claiming DynamoDB will be holding full 20TB in RAM if yes I have a bridge to sell you :). The price for i3.16xlarge was quoted for 2 instances the IOPS used for calculation is 1/8th of actual max IOPS that instance can do.

> Did I ever claim on 48TB on AWS

When presenting an alternative to DynamoDB, AWS is an implied part of presenting that alternative. While the original article does suggest an alternative outside of AWS, it properly qualifies it as a competitor offering. Regardless, I don't care what was and wasn't claimed. It's not important.

> You are seriously claiming DynamoDB will be holding full 20TB in RAM

No. I'm claiming that DynamoDB will have predictable and consistent latency characteristics at 20TB. The related claim is that an RDBMS will not have consistent latency characteristics if the working dataset does not fit in memory.

Re: There are very few suitable use cases for DynamoDB

#136

Post by qaq »

phamilton wrote: post #135
qaq wrote: post #134

Earlier quoted context omitted.

Ok let's try :) Did I ever claim on 48TB on AWS ? You are seriously claiming DynamoDB will be holding full 20TB in RAM if yes I have a bridge to sell you :). The price for i3.16xlarge was quoted for 2 instances the IOPS used for calculation is 1/8th of actual max IOPS that instance can do.

> Did I ever claim on 48TB on AWS When presenting an alternative to DynamoDB, AWS is an implied part of presenting that alternative. While the original article does suggest an alternative outside of AWS, it properly qualifies it as a competitor offering. Regardless, I don't care what was and wasn't claimed. It's not important. > You are seriously claiming DynamoDB will be holding full 20TB in RAM No. I'm claiming tha…

"When presenting an alternative to DynamoDB, AWS is an implied part of presenting that alternative". If that is the way you guys operate well ... We run things in AWS, on prem and in GCP. Our AWS spend is over 3 mil. per month things that require hard latency guarantees are def. not running in AWS. Nothing is AWS has "predictable and consistent latency characteristics" simply due to the nature of how they operate.

Re: There are very few suitable use cases for DynamoDB

#137

Post by ckalantzis »

pishpash wrote: post #108
dserban wrote: post #101

My background is in Cassandra and one company I worked for last year insisted that we use DynamoDB for a project. Here are a few things that ended up being show stoppers. 1. Both the partition key and the sort key are capped at 1 field. In an attempt to "think Cassandra data model", the ugly workaround was to stringify and concatenate things at the application layer, then parse / split on the other side. This made th…

Cassandra isn't any better in those regards. For 1, Cassandra just does the concatenation under the hood and 3 happens all the time on Cassandra. The broken thing is using NoSQL as a DB, not Cassandra or DynamoDB.

Quick! Someone should tell Netflix and Apple, that. They've been using Cassandra and Datastax as their main DB for almost a decade.

They're always hiring. You should apply and show them the error of their ways.

Re: There are very few suitable use cases for DynamoDB

#138

Post by linuxhansl »

dijit wrote: post #96
threeseed wrote: post #42

Earlier quoted context omitted.

You are talking complete nonsense. HBase, Cassandra, MongoDB, Riak, Couchbase etc all write through to disk with proper fsync flushes. And I've never heard of any database that has a model where it writes to a virtual file system - whatever that even means. Please provide some specific examples.

HBase, Cassandra, MongoDB, Riak, Couchbase And redis were the main candidates. Hbase: http://mail-archives.apache.org/mod_mbox/hbase-issues/201307... Cassandra: (fsync to WAL, not full fsync). https://wiki.apache.org/cassandra/Durability MongoDB: ... too much wrong here to list, although I hear it's improving in being cluster aware etc. Redis does support fsync as far as I remember but the write/delete pattern is inc…

As author of the cited post on HBase lemme clarify:

0. Here are more details on that: http://hadoop-hbase.blogspot.de/2013/07/protected-hbase-agai...

1. By default HBase "flushes" the WAL. Flush here means to make sure that at least 3 machines have change in memory (NOT on disk). A datacenter power outage can lose data.

2. As HDFS closes a block it is not by default forced to disk. So as HBase rewrites old data during compactions by default, old data can be lost during a power outage. Again, by default.

3. HDFS should be configured with sync-on-close, so that old data is forced to disk upon compactions (and sync-behind-writes for performance)

4. HBase now has an option to force and a WAL edit (and all previous edits) to disk (that's what I added in said jira).

5. This is post is 4 years old for chrissake :)... Don't base decisions on 4 year old information.

HBase _is_ a database and it will keep your data safe. Unfortunately it requires some configuration and some knowledge.

Re: There are very few suitable use cases for DynamoDB

#139

Post by pzb »

phamilton wrote: post #73
throwaway91111 wrote: post #65

Earlier quoted context omitted.

I did. However, 10GB still seems extremely small. A commodity postgres, cassandra, or cockroachdb server can serve HUNDREDS of GB per node. Why is the size per node so small for dynamodb? It seems like poor key space design.

While not exactly the same, the Dynamo paper outlines that a single host is composed of many virtual nodes. It is very likely that a physical DynamoDB host will have dozens of nodes. This is done so that the cluster can scale up or down independent of the number of hosts while avoiding a gross imbalance. (12 nodes on 11 hosts means one host has 100% more traffic. 34 nodes on 11 hosts means one host has 33% more traff…

It is important to note that "Dynamo" and "DynamoDB" are two very different things that happen to share many of the same letters. DynamoDB is not Dynamo.

Re: There are very few suitable use cases for DynamoDB

#140

Post by scrollaway »

dijit wrote: post #4
jsemrau wrote: post #2

When I started building my first app in 2011 MongoDB was the rage. So I build the back-end using the futuristic 'No-SQL' technology. It turned out to be slow (~1 min Query time), inconsistent, and missing an RDBMS layer. Move the thing to PHP/Mysql problems were gone. I still have not found a use case outside of web (comments/discussion) sites where the high integration with Javascript actually makes sense.

I hope you don't mind if I piggyback on this to echo this sentiment. Although I dislike MySQL for its many gotchas (data corruption level stuff too!) I was looking for a _long_ _long_ time for a high consistency NoSQL database.. we basically need document storage of large binary data. Ironically literally nothing in NoSQL land does write-through to disk, they just write to vfs and hope it works; additionally, those t…

Key-value document storage of large binary data... isn't that S3?
Post reply on HN