Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

ZooKeeper vs. Doozer vs. Etcd

devo.ps

Post reply on HN
Print view View original

21–30 of 91 posts

Re: ZooKeeper vs. Doozer vs. Etcd

#21

Post by tomjohnson3 »

russell_h wrote: post #13

One point the article didn't cover is clients. Making a good Zookeeper client is hard, for two reasons: 1. The protocol is difficult to implement. In theory you could just use Jute to codegen this part, but that assumes Jute supports the language you need. Doozer improves on this with a simple text-based protocol, and etcd goes a little further with a HTTP API. 2. The primitives Zookeeper exposes are very primitive.…

one other thing to keep in mind is that the underlying algorithms (zab and raft) provide different guarantees.

for example, zookeeper/zab allows reading directly from followers with a guarantee to get at least a past value that won't be rolled back. this was one reason zookeer didn't use paxos:

https://cwiki.apache.org/confluence/display/ZOOKEEPER/Zab+vs...

in my understanding, raft doesn't allow reading directly from followers, because the follower logs may get repaired/rolled-back when a new leader is elected. (though i'm sure an implementation can tweak the protocol to provide this support.)

that said, raft has a lot of interesting applications, and, in my opinion, is definitely more understandable than the many versions of paxos. (implementing zab yourself, at this point, would be a futile exercise.)

i found the videos from the raft user study to be very well done (and easier to understand than even their paper):

raft: http://www.youtube.com/watch?v=JEpsBg0AO6o paxos: http://www.youtube.com/watch?v=YbZ3zDzDnrw

...however, i think they did paxos a disadvantage by not just focusing on multi-paxos (which is probably the most common implementation). but, it's certainly fair to say that info about paxos is spread out far and wide...with perhaps too many knobs to turn and implementation-related details to fill in yourself.

as a side note: i've just started implementing raft in a set of libraries (multiple languages) that will be open source - along with other protocols.

Re: ZooKeeper vs. Doozer vs. Etcd

#22

Post by johnbellone »

sandfox wrote: post #19
johnbellone wrote: post #8

Earlier quoted context omitted.

Even though Zookeeper is "mature" it is definitely a beast to fine tune and wade through. Zookeeper definitely does not bring along a ton of dependencies. If you're deploying it as documented it should be running on its own dedicated machines. I don't see why dependencies are a problem there especially with Chef/Puppet nowadays.

Just because you have tools ease dealing with dependencies, it does not mean that those dependencies and the cost of dealing with them has gone away. Easier to not have the problem instead of solving it...

What exactly are there problems with? I am behind a strict corporate network and all I needed to do was download the Zookeeper JAR and write a couple of deployment scripts. Even though the environment variables for the startup scripts are lacking they cover nearly everything you can want.

Zookeeper has legitimate issues and from experience most of which stems from the documentation being so verbose that it takes a lot of fine tuning to get right. But if you put it on its own hardware, or at the very least, the write-ahead log on its own partition, you should be good go.

Re: ZooKeeper vs. Doozer vs. Etcd

#23

Post by alecthomas »

The story of Doozer is a classic example of how not to steward an open source project.

It was released by two Heroku engineers who promptly completely abandoned it. By completely I mean did not respond to any communication whatsoever for a year or so, despite a very active community that had sprung up around the project in terms of users and forks. I don't begrudge them their lives (or whatever drew them away), but there were people/companies willing and able to take over maintenance, but not even that happened. It probably would have taken just a few hours to hand over maintainership. Instead, just radio silence.

Eventually there was some movement and the most active group of fork maintainers were given commit access, but by that time any enthusiasm over Doozer was long dead and gone.

Re: ZooKeeper vs. Doozer vs. Etcd

#24

Post by voidmain »

russell_h wrote: post #13

One point the article didn't cover is clients. Making a good Zookeeper client is hard, for two reasons: 1. The protocol is difficult to implement. In theory you could just use Jute to codegen this part, but that assumes Jute supports the language you need. Doozer improves on this with a simple text-based protocol, and etcd goes a little further with a HTTP API. 2. The primitives Zookeeper exposes are very primitive.…

I agree. At FoundationDB, we're writing a coordination tool (working name "beastmaster") that provides service discovery, locking, leader election, etc on top of our transactional key/value store. It is higher level than these tools; our idea is to try to make it useful from a command line or DNS rather than have to be baked into every piece of software that needs service discovery.

    beast service --lock name=mailserver.foundationdb.com --run mailserver.sh &
    beast service name=webserver.foundationdb.com --run webserver.sh &
    ping mailserver.foundationdb.com
    wget http://webserver.foundationdb.com/

Re: ZooKeeper vs. Doozer vs. Etcd

#25

Post by hunvreus »

sandfox wrote: post #20
alanctgardner2 wrote: post #9

Earlier quoted context omitted.

Exactly. A "devops" company shouldn't really be complaining about dependencies anyways, they're a fact of life. I admit my experience with ZK has been 50% administering it with Cloudera Manager, which basically abstracts away all the nastiness. I'm curious about specific issues people have had though; I've used ZK with Hadoop and Kafka without any major issues.

Why shouldn't a 'devops' company complain about dependency? It's their domain completely. Treating anything as a "fact of life" is not going to make the problem go away... And what about anyone out there trying to use them who isn't a 'devops' thingy. A painful dependency graph is going to be something that influences their decision on what tool they choose.

Pretty much what I thought. The leaner the better.

Moreover, in our specific case, we did not want to introduce dependencies on hosts that are managed by our customers so as to not run into conflicts with their own stack.

Re: ZooKeeper vs. Doozer vs. Etcd

#26

Post by enigmo »

voidmain wrote: post #24
russell_h wrote: post #13

One point the article didn't cover is clients. Making a good Zookeeper client is hard, for two reasons: 1. The protocol is difficult to implement. In theory you could just use Jute to codegen this part, but that assumes Jute supports the language you need. Doozer improves on this with a simple text-based protocol, and etcd goes a little further with a HTTP API. 2. The primitives Zookeeper exposes are very primitive.…

I agree. At FoundationDB, we're writing a coordination tool (working name "beastmaster") that provides service discovery, locking, leader election, etc on top of our transactional key/value store. It is higher level than these tools; our idea is to try to make it useful from a command line or DNS rather than have to be baked into every piece of software that needs service discovery. beast service --lock name=mailserv…

That's how our Chubby clone works... it uses the Route53 API to publish service endpoints and leaders.

Re: ZooKeeper vs. Doozer vs. Etcd

#27

Post by enigmo »

darkarmani wrote: post #18
alanctgardner2 wrote: post #7

Earlier quoted context omitted.

Oh! Treat records as key-value pairs, and encode the value into the IP. If your timeout is 10 seconds, add a DNS record timeout.server.yourdomain which resolves to an IPv6 address with value 10. It gets tougher with ASCII strings, but you could support multi-record configs as well. Then your application just uses nslookup to download the config when you reload it. If someone builds this I will be their best friend

You can also use DNS like a distributed cache. This is useful when you have millions of clients because their local DNS server will do caching for you. You can also use it like a bloom filter where cache hits are true positives and anything that misses might or might not be a valid key.

That doesn't sound much at all like a bloom filter.

Re: ZooKeeper vs. Doozer vs. Etcd

#28

Post by hunvreus »

alanctgardner2 wrote: post #14
hunvreus wrote: post #10

Earlier quoted context omitted.

As we said, we don't hate Java. We were however constrained in what we could install on the host machines since we administrate these on behalf of others. Even lightweight dependencies were a no-go for us. With regards to Apache, we have "mixed feelings". We're aligned with a lot of what they stand for and are definitely very thankful for what they enabled in the OSS community, but are not enthused with the way they…

I get the mixed Apache feelings, but I still don't quite get the dependencies complaint. It seems like your clients have heterogeneous environments (multiple Linux distros, at least). Do you distribute a statically linked binary, or do they compile it on their box? It seems like in this case Go code is actually way more complex to set up.

We actually deal with a limited number of distros for now (mostly Ubuntu and RHE) and we distribute statically linked binaries (meaning no dependency at all).

Re: ZooKeeper vs. Doozer vs. Etcd

#29

Post by joevandyk »

I use postgresql + listen/notify to share and push configuration out to applications.

Each application starts a thread that LISTEN's for NOTIFY's from postgresql.

I have a settings table (name text, value text). Configuration data is stored there.

    insert into settings (name, value) values
      ('sites.my-site.authnet.api_key', 'asdfasdf');
There's a trigger on that table that issues a NOTIFY to all the clients. When the clients receive the NOTIFY, they query the table and store all of the settings in memory.

It works great.

No additional moving parts. All my configuration is stored in the database, they aren't checked into files that need to be protected. No security concerns about api keys stored in a separate service. My settings are backed up along with the rest of my data.

Re: ZooKeeper vs. Doozer vs. Etcd

#30

Post by Juha »

alecthomas wrote: post #23

The story of Doozer is a classic example of how not to steward an open source project. It was released by two Heroku engineers who promptly completely abandoned it. By completely I mean did not respond to any communication whatsoever for a year or so, despite a very active community that had sprung up around the project in terms of users and forks. I don't begrudge them their lives (or whatever drew them away), but t…

I agree. It's a pity Doozer got to this point. I was happy to see that still this years Febrary proactive developers in the google groups (https://groups.google.com/forum/#!topic/doozer/fVcS0y3KuHQ) tried to get the project active and coordinated, but I guess merging different codebases from different forks was too big of a challenge.
Post reply on HN