Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Dynamo Systems Work Too Hard

damienkatz.net

Post reply on HN
Print view View original

11–20 of 29 posts

Re: Dynamo Systems Work Too Hard

#11

Post by threeseed »

damienkatz wrote: post #8
jbellis wrote: post #4

This misses the point. There are two main reasons why, when I was researching scalable databases, I primarily gravitated towards Dynamo-style replication (Cassandra, Voldemort, and at the time, Dynomite): - There is no such thing as failover. Dynamo replication takes node failure in stride. This is what you want for a robust system where "Network Partitions are Rare, Server Failures are Not." Not only does it prevent…

I'm not denying that Dynamo features more consistent availability, but it does so at either cost of temporal consistency or a much larger amount of resources. That's the point of the article, the tradeoffs are expensive and you can achieve effectively the same performance and availability with fewer resources. Dynamo may rule out a certain class of bugs, but that doesn't mean other systems must also have those bugs.…

Can you possibly be a bit more specific when talking about 'resources' and 'much larger'. Makes it hard to evaluate if it is even an issue or not.

Also can you elaborate more on the multi-DC part of CouchBase. Documentation seems to imply only supports two-DC replication and I find the "currently supports continuous replication of data" line to be odd. What are the future plans for CouchBase in this area ?

Re: Dynamo Systems Work Too Hard

#13

Post by gukjoon »

Great article, Damien. This idea that network partitions are exceedingly rare was the reason why ElasticSearch goes CA vs. the AP many other NoSQL datastores choose.

http://elasticsearch-users.115913.n3.nabble.com/CAP-theorem-...

Not only are network partitions rare, the most disastrous case where the cluster splits in half is even rarer. Usually, you have a small part of the cluster partition away.

I hope people don't take this as a Dynamo vs. Couch discussion, because the relative importance of partition tolerance is a topic that spans all datastores that give up on ACID.

Re: Dynamo Systems Work Too Hard

#14

Post by strlen »

damienkatz wrote: post #8
jbellis wrote: post #4

This misses the point. There are two main reasons why, when I was researching scalable databases, I primarily gravitated towards Dynamo-style replication (Cassandra, Voldemort, and at the time, Dynomite): - There is no such thing as failover. Dynamo replication takes node failure in stride. This is what you want for a robust system where "Network Partitions are Rare, Server Failures are Not." Not only does it prevent…

I'm not denying that Dynamo features more consistent availability, but it does so at either cost of temporal consistency or a much larger amount of resources. That's the point of the article, the tradeoffs are expensive and you can achieve effectively the same performance and availability with fewer resources. Dynamo may rule out a certain class of bugs, but that doesn't mean other systems must also have those bugs.…

> I'm not denying that Dynamo features more consistent availability, but it does so at either cost of temporal consistency or a much larger amount of resources.

The trouble is that your article outlines problems with pretty much all quorum systems, including multi-Paxos: the setup where there's an elected leader -- elected by first round of Paxos -- which then performs subsequent writes using a single-round-trip second-round.

Incidentally, that is very close to what you've proposed, except you've chosen to not perform quorum reads. That is perfectly fine, but there are also hidden costs -- not only do you need to have a leader election, but you also need synchronization barriers. Before a node can handle writes or serve latest reads, it must also back-fill its transaction up to the last entry (which it does by reading from peers).

So while you are saving on read traffic (online reads only go to the master), you are now decreasing availability (contrary to your stated goal), and increasing system complexity.

You also do hurt performance by requiring all writes and reads to be serialized through a single node: unless you plan to have a leader election whenever the node fails to meet a read SLA (which is going to result a disaster -- I am speaking from personal experience), you will have to accept that you're bottlenecked by a single node. With a Dynamo-style quorum (for either reads or writes), a single straggler will not reduce whole-cluster latency.

The core point of Dynamo is low latency, availability and handling of all kinds of partitions: whether clean partitions (long term single node failures), transient failures (garbage collection pauses, slow disks, network blips, etc...), or even more complex dependent failures.

The reality, of course, is that availability is neither the sole, nor the principal concern of every system. It's perfect fine to trade off availability for other goals -- you just need to be aware of that trade off.

You may want to note evolution of Google's services: BigTable -> MegaStore -> Spanner. Essentially, they've started out with a non-HA system (BigTable), found that every team has began implement HA on their own, and then added quorum-based protocols (Paxos), finally adding an optimization (TrueTime) to reduce read-latency.

Re: Dynamo Systems Work Too Hard

#15

Post by gigq »

This also exactly describes how HBase works. I've always preferred HBase to Cassandra for this exact reason. You put far less read load on your servers and you don't have to worry about most of the things on http://wiki.apache.org/cassandra/Operations.

Another benefit that is not mentioned is that with a master based system you can easily move who is responsible for the data if a server starts to hotspot. In Cassandra you have to use random key distribution because if you have a server hotspot then the only solution is to split the token ring which is an intensive operation that is hard to do while the server is under heavy load.

Re: Dynamo Systems Work Too Hard

#16

Post by damienkatz »

pbailis wrote: post #10

There's at least one good reason for Dynamo's write-to-all and read-from-all mechanism: latency. What you've called 'W=2' in Couchbase is "write to master and at least one slave." Dynamo-style 'W=2' means "write to any two replicas." This can decrease tail latencies since you don't have to wait for the master--any two will do; similarly for 'R=2'. Indeed, Dynamo 'W=2, R=2' will incur more read load than master-based…

Good point. But "writes" are very fast, in our tests write latency is less than half read latency, so we can easily do master to slave replication within the SLA. But you point is correct, a Dynamo system is faster to achieve the same replication factor.

Re: Dynamo Systems Work Too Hard

#17

Post by mrphoebs »

gigq wrote: post #15

This also exactly describes how HBase works. I've always preferred HBase to Cassandra for this exact reason. You put far less read load on your servers and you don't have to worry about most of the things on http://wiki.apache.org/cassandra/Operations . Another benefit that is not mentioned is that with a master based system you can easily move who is responsible for the data if a server starts to hotspot. In Cassand…

You could always just do a read=any on a cassandra cluster.

Re: Dynamo Systems Work Too Hard

#18

Post by jbellis »

damienkatz wrote: post #16
pbailis wrote: post #10

There's at least one good reason for Dynamo's write-to-all and read-from-all mechanism: latency. What you've called 'W=2' in Couchbase is "write to master and at least one slave." Dynamo-style 'W=2' means "write to any two replicas." This can decrease tail latencies since you don't have to wait for the master--any two will do; similarly for 'R=2'. Indeed, Dynamo 'W=2, R=2' will incur more read load than master-based…

Good point. But "writes" are very fast, in our tests write latency is less than half read latency, so we can easily do master to slave replication within the SLA. But you point is correct, a Dynamo system is faster to achieve the same replication factor.

I think it's worth elaborating that the primary advantage to the dynamo model is not in the best- or average- case, but when everything does not go as planned -- when the master gets behind, when ec2 network latency spikes, etc. Then "any two replicas" instead of "master plus one more" is much more robust.

Re: Dynamo Systems Work Too Hard

#20

Post by linuxhansl »

gigq wrote: post #15

This also exactly describes how HBase works. I've always preferred HBase to Cassandra for this exact reason. You put far less read load on your servers and you don't have to worry about most of the things on http://wiki.apache.org/cassandra/Operations . Another benefit that is not mentioned is that with a master based system you can easily move who is responsible for the data if a server starts to hotspot. In Cassand…

Yep.

I hear the "master design is bad" argument all the time.

From many angles it is a bad design, but from other's it is not. First off, it is simpler and easier grok and check for bugs. Debugging a running system is easier.

It is also easier to implement different distribution strategies and failure/placementgroups, because that algorithms is centralized. If things go wrong it is easier to track where your data is.

No gossip rings to converge. And it is easier to grow the cluster.

/HBase committer here.

Post reply on HN