Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Dynamo Systems Work Too Hard

damienkatz.net

Post reply on HN
Print view View original

21–29 of 29 posts

Re: Dynamo Systems Work Too Hard

#21

Post by banachtarski »

damienkatz wrote: post #16
pbailis wrote: post #10

There's at least one good reason for Dynamo's write-to-all and read-from-all mechanism: latency. What you've called 'W=2' in Couchbase is "write to master and at least one slave." Dynamo-style 'W=2' means "write to any two replicas." This can decrease tail latencies since you don't have to wait for the master--any two will do; similarly for 'R=2'. Indeed, Dynamo 'W=2, R=2' will incur more read load than master-based…

Good point. But "writes" are very fast, in our tests write latency is less than half read latency, so we can easily do master to slave replication within the SLA. But you point is correct, a Dynamo system is faster to achieve the same replication factor.

Do you surround the word "writes" in quotes because they aren't truly writes in the durable sense?

Re: Dynamo Systems Work Too Hard

#22

Post by banachtarski »

damienkatz wrote: post #8
jbellis wrote: post #4

This misses the point. There are two main reasons why, when I was researching scalable databases, I primarily gravitated towards Dynamo-style replication (Cassandra, Voldemort, and at the time, Dynomite): - There is no such thing as failover. Dynamo replication takes node failure in stride. This is what you want for a robust system where "Network Partitions are Rare, Server Failures are Not." Not only does it prevent…

I'm not denying that Dynamo features more consistent availability, but it does so at either cost of temporal consistency or a much larger amount of resources. That's the point of the article, the tradeoffs are expensive and you can achieve effectively the same performance and availability with fewer resources. Dynamo may rule out a certain class of bugs, but that doesn't mean other systems must also have those bugs.…

Temporal consistency is overrated for many applications and still more applications achieve consistency with a locking mechanism or serializing reads and writes elsewhere.

As for resources, I've heard it say that your architecture is working for you if you can throw more resources at the problem, and it scales linearly. If I have a key that gets accessed 2000/s on a single Couchbase node, I'm going to get long tails on every request that goes to that node.

Re: Dynamo Systems Work Too Hard

#23

Post by bscanlan »

leef wrote: post #12

Despite the name you can't actually assume DynamoDB is based on the Dynamo paper architecture.

"The original Dynamo design was based on a core set of strong distributed systems principles resulting in an ultra-scalable and highly reliable database system. Amazon DynamoDB, which is a new service, continues to build on these principles, and also builds on our years of experience with running non-relational databases and cloud services, such as Amazon SimpleDB and Amazon S3, at scale."

http://www.allthingsdistributed.com/2012/01/amazon-dynamodb....

Re: Dynamo Systems Work Too Hard

#24

Post by bscanlan »

crb wrote: post #3

When your units of networking concern are "availability zones" (i.e. data centers) rather than just switches, wouldn't network failures now be more common than server failures?

Agreed - the assumption is that the MTBF of a single network device is a relevant statistic is undoubtedly incorrect in anywhere but a hobbyist's network.

Re: Dynamo Systems Work Too Hard

#25

Post by pkolaczk »

gukjoon wrote: post #13

Great article, Damien. This idea that network partitions are exceedingly rare was the reason why ElasticSearch goes CA vs. the AP many other NoSQL datastores choose. http://elasticsearch-users.115913.n3.nabble.com/CAP-theorem-... Not only are network partitions rare, the most disastrous case where the cluster splits in half is even rarer. Usually, you have a small part of the cluster partition away. I hope people don…

Network partitions are extremeley rare only for small clusters. For very large clusters or multi-datacenter clusters there is much more hardware than a single switch between servers. Then, the likelihood that something cuts off the whole room full of servers from the rest of your cluster is not something safe to neglect.

Re: Dynamo Systems Work Too Hard

#26

Post by pkolaczk »

Why compare MTBF of a single network switch to MTBF of a node? Why not compare MTBF of a single network switch to MTBF of a single CPU or a motherboard? Unless you're talking about hobby-size network, there is usually much more between the nodes than a single network switch.

Re: Dynamo Systems Work Too Hard

#27

Post by linuxhansl »

linuxhansl wrote: post #20
gigq wrote: post #15

This also exactly describes how HBase works. I've always preferred HBase to Cassandra for this exact reason. You put far less read load on your servers and you don't have to worry about most of the things on http://wiki.apache.org/cassandra/Operations . Another benefit that is not mentioned is that with a master based system you can easily move who is responsible for the data if a server starts to hotspot. In Cassand…

Yep. I hear the "master design is bad" argument all the time. From many angles it is a bad design, but from other's it is not. First off, it is simpler and easier grok and check for bugs. Debugging a running system is easier. It is also easier to implement different distribution strategies and failure/placementgroups, because that algorithms is centralized. If things go wrong it is easier to track where your data is.…

Thanks for the down vote.

If you find any technical inconsistencies in what I said, please let me know.

Re: Dynamo Systems Work Too Hard

#28

Post by IGuardBridges »

strlen wrote: post #14
damienkatz wrote: post #8

Earlier quoted context omitted.

I'm not denying that Dynamo features more consistent availability, but it does so at either cost of temporal consistency or a much larger amount of resources. That's the point of the article, the tradeoffs are expensive and you can achieve effectively the same performance and availability with fewer resources. Dynamo may rule out a certain class of bugs, but that doesn't mean other systems must also have those bugs.…

> I'm not denying that Dynamo features more consistent availability, but it does so at either cost of temporal consistency or a much larger amount of resources. The trouble is that your article outlines problems with pretty much all quorum systems, including multi-Paxos: the setup where there's an elected leader -- elected by first round of Paxos -- which then performs subsequent writes using a single-round-trip seco…

One quick point - AFAIK Megastore uses quorum based across data centers but uses BigTable within a data center - so master based. Is that not true? If it is then your point does need to be qualified - cross center DC failure models are (I assume) different from those within a DC.

Re: Dynamo Systems Work Too Hard

#29

Post by gigq »

mrphoebs wrote: post #17
gigq wrote: post #15

This also exactly describes how HBase works. I've always preferred HBase to Cassandra for this exact reason. You put far less read load on your servers and you don't have to worry about most of the things on http://wiki.apache.org/cassandra/Operations . Another benefit that is not mentioned is that with a master based system you can easily move who is responsible for the data if a server starts to hotspot. In Cassand…

You could always just do a read=any on a cassandra cluster.

There are a few problems with read=any (or really read=one as any is only supported by writes). One issue is it will still send the request out to all the servers in the quorum so even if you don't have to wait on a quorum you still put the read load on them.

The second and more important issue is that if you want consistent data the only way to use read=one is if you use write=all which means you would have no resilience to a server outage. So in a normal cassandra cluster where you have write=quorum a read=one can give you back no data. In my experience this happened frequently enough with long GC pauses on one of the nodes that it wasn't useful to use in production.

Post reply on HN