Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Consul, a new tool for service discovery and configuration

hashicorp.com

Post reply on HN
Print view View original

41–50 of 62 posts

Re: Consul, a new tool for service discovery and configuration

#41

Post by armon »

opendais wrote: post #20
hardwaresofton wrote: post #16

Earlier quoted context omitted.

So I think here the problem isn't really the datastore, it's more the high-availability and discovery. The main bonus that Consul seems to be providing is maintaining a logical topology of your network without you doing much. They do this by a using gossip-based protocol and a derivative of paxos called Raft. These two things work together to essentially have the servers that run your various services (whether api or…

Wouldn't a centralized Cassandra cluster be reliable enough to meet that need? I've never had a cluster completely collapse on me unless things were already screwed up enough that Service Discovery was ultimately useless since nothing else would work. It just seems to me that losing your datastore makes your services unusable...at which point 'discovering them' isn't really the issue. Instead, everyone wants to intro…

To reiterate a bit on what hardwaresofton has already said, yes you could use Cassandra. However, Cassandra like ZooKeeper is just a building block. It doesn't actually provide service discovery, health checking, etc. You could use it to build all of those, but it doesn't provide it.

We compare Consul to ZooKeeper here, but much of that applies to Cassandra as well: http://www.consul.io/intro/vs/zookeeper.html

Internally, Consul could also use something like Cassandra to store the data. However we use LMDB which is an embedded database to avoid an expensive context switch out of the process to serve requests with lower latency and higher throughput.

Re: Consul, a new tool for service discovery and configuration

#42

Post by mitchellh »

peterwwillis wrote: post #37
mitchellh wrote: post #34

Earlier quoted context omitted.

Thanks for prefacing the constructive criticism with the compliment. :) I want to clarify: Of all the buzz words Consul has, one thing Consul ISN'T is decentralized. You must run at least one Consul server in a cluster. If you want a fully centralized approach, you can just run one server. No big deal. Of course, if that server goes down, reads/writes are unavailable. If you want high availability, you run multiple s…

Oh, well i'm a giant ass then. I saw "completely distributed" and extrapolated decentralized from that. Still, an automatically-[randomly?]-elected leader in a distributed network is a form of decentralized network. It's effectively a toroidal network topology, since your datacenters make it multi-dimensional (though now that I look at hypercubes i'm not exactly sure which is a better fit here) Now that i've re-read…

We recommend three or five because that will get you a high level of fault tolerance. Consul runs just fine with one server, but you'll have no fault tolerance: if that one server becomes unavailable for any reason, then reads/writes for your cluster will not be served.

The leader is elected using Raft as the consensus algorithm (it is linked in the internals section of the docs).

You are correct though that if you run multiple servers, the leader election will happen automatically. There is no way to manually override it.

Re: Consul, a new tool for service discovery and configuration

#43

Post by opendais »

hardwaresofton wrote: post #36
opendais wrote: post #30

Earlier quoted context omitted.

I wasn't trying to start a complex argument in someone else's thread. Sorry. Maybe we should drop it and/or discuss elsewhere? I think you misunderstood what I was talking about based on your explanation with a single physical machine running a single database instance and manually adding nodes requiring human intervention.

Ah OK, was feeling I was explaining far too much -- no worries, but if HN is no longer a place for complex arguments, I think it'd be a lot less valuable... Also, me being wrong certainly sounds like a great chance for Hashicorp people to correct me.

You were looking at very, very old documentation which is why I wanted to drop it. Cassandra is on 2.0 and uses a practically identical consensus algorithm [Paxos] to Consul [Raft] except for the implementation details. I'm not sure how much practical advantage something like Consul has over a discovery layer built on Cassandra.

I didn't see a good way to sort this conversation out without a very, very long conversation in a thread dedicated to a release announcement.

Re: Consul, a new tool for service discovery and configuration

#44

Post by opendais »

armon wrote: post #41
opendais wrote: post #20

Earlier quoted context omitted.

Wouldn't a centralized Cassandra cluster be reliable enough to meet that need? I've never had a cluster completely collapse on me unless things were already screwed up enough that Service Discovery was ultimately useless since nothing else would work. It just seems to me that losing your datastore makes your services unusable...at which point 'discovering them' isn't really the issue. Instead, everyone wants to intro…

To reiterate a bit on what hardwaresofton has already said, yes you could use Cassandra. However, Cassandra like ZooKeeper is just a building block. It doesn't actually provide service discovery, health checking, etc. You could use it to build all of those, but it doesn't provide it. We compare Consul to ZooKeeper here, but much of that applies to Cassandra as well: http://www.consul.io/intro/vs/zookeeper.html Intern…

This is more of a wishlist item in my professional life. I'm not trying to dispute the choice you made which is to add an additional type of critical cluster for persistence purposes, since that is how all of such services I've seen so far are designed.

I just wish I could avoid having to maintain Cluster Type A for service discovery and Cluster Type B for data storage.

Re: Consul, a new tool for service discovery and configuration

#45

Post by stormbrew »

So I'm mostly curious why this isn't just basically serf 2.0. Looking at serf I never really felt like it had much use in the basic form it took, with no ability to advertise extra details about the nodes in a dynamic fashion. Consul seems to build onto serf the things that serf needed to become really useful, so seems more like a successor to serf than a parallel project.

It seems like the right thing to do here would be to take the lessons of building consul into making serf something more like a library on which to build other things rather than a service in its own right.

Re: Consul, a new tool for service discovery and configuration

#46

Post by anonymouse01 »

opendais wrote: post #43
hardwaresofton wrote: post #36

Earlier quoted context omitted.

Ah OK, was feeling I was explaining far too much -- no worries, but if HN is no longer a place for complex arguments, I think it'd be a lot less valuable... Also, me being wrong certainly sounds like a great chance for Hashicorp people to correct me.

You were looking at very, very old documentation which is why I wanted to drop it. Cassandra is on 2.0 and uses a practically identical consensus algorithm [Paxos] to Consul [Raft] except for the implementation details. I'm not sure how much practical advantage something like Consul has over a discovery layer built on Cassandra. I didn't see a good way to sort this conversation out without a very, very long conversat…

Consul is a totally different thing to Cassandra. Consul could be built on top of Cassandra, but it would not provide any advantages. Why would you want to maintain a separate datastore?

Consul is a single static binary that uses around 11MB of RAM. Cassandra is a complex distributed datastore that requires the JVM and uses many gigabytes of RAM.

Re: Consul, a new tool for service discovery and configuration

#47

Post by opendais »

anonymouse01 wrote: post #46
opendais wrote: post #43

Earlier quoted context omitted.

You were looking at very, very old documentation which is why I wanted to drop it. Cassandra is on 2.0 and uses a practically identical consensus algorithm [Paxos] to Consul [Raft] except for the implementation details. I'm not sure how much practical advantage something like Consul has over a discovery layer built on Cassandra. I didn't see a good way to sort this conversation out without a very, very long conversat…

Consul is a totally different thing to Cassandra. Consul could be built on top of Cassandra, but it would not provide any advantages. Why would you want to maintain a separate datastore? Consul is a single static binary that uses around 11MB of RAM. Cassandra is a complex distributed datastore that requires the JVM and uses many gigabytes of RAM.

Why would you want to maintain a separate service discovery cluster?

Re: Consul, a new tool for service discovery and configuration

#48

Post by armon »

stormbrew wrote: post #45

So I'm mostly curious why this isn't just basically serf 2.0. Looking at serf I never really felt like it had much use in the basic form it took, with no ability to advertise extra details about the nodes in a dynamic fashion. Consul seems to build onto serf the things that serf needed to become really useful, so seems more like a successor to serf than a parallel project. It seems like the right thing to do here wou…

The two offer very different trade offs and have different architectures. Serf is an AP system, meaning it trades of consistency for availability. 90% of your nodes can be down and a Serf cluster will continue to operate. It does offer a primitive set of features, but it is useful for a wide variety of tasks such as configuring a memcache pool, load balancers, P2P VPN topologies, etc.

Consul is a CP system, meaning it trades availability for consistency. It has a much more limited ability to tolerate failures. However, its more central architecture allows it to support a richer feature set.

By keeping the tools separate we give developers and operators two different tools. Sometimes you need a hammer, and sometimes a screwdriver will do.

This page compares the two: http://www.consul.io/intro/vs/serf.html

Re: Consul, a new tool for service discovery and configuration

#49

Post by mitchellh »

stormbrew wrote: post #45

So I'm mostly curious why this isn't just basically serf 2.0. Looking at serf I never really felt like it had much use in the basic form it took, with no ability to advertise extra details about the nodes in a dynamic fashion. Consul seems to build onto serf the things that serf needed to become really useful, so seems more like a successor to serf than a parallel project. It seems like the right thing to do here wou…

To add to what Armon said: we always had Consul in mind when building Serf. But Serf was a necessary building block on the way to Consul: highly available, lightweight membership management. Every distributed system has a membership problem, and we didn't want to reinvent the wheel for every system we built. Serf provided an incredibly stable and well proven foundation for us to build Consul on top, as well as future products that are already in the works...

And while you may not see Serf as having much use, we've personally helped and seen Serf clusters with many thousands of nodes. Serf is very useful to these organizations for its purpose. And while some of these orgs are now looking at Consul, many don't need Consul in the same way (but may deploy it separately).

We're not stopping with Consul. We have something more on the way. But we now have some great building blocks and experience building distributed systems to keep doing it correctly without having to rebuild everything from scratch.

Re: Consul, a new tool for service discovery and configuration

#50

Post by stormbrew »

mitchellh wrote: post #49
stormbrew wrote: post #45

So I'm mostly curious why this isn't just basically serf 2.0. Looking at serf I never really felt like it had much use in the basic form it took, with no ability to advertise extra details about the nodes in a dynamic fashion. Consul seems to build onto serf the things that serf needed to become really useful, so seems more like a successor to serf than a parallel project. It seems like the right thing to do here wou…

To add to what Armon said: we always had Consul in mind when building Serf. But Serf was a necessary building block on the way to Consul: highly available, lightweight membership management. Every distributed system has a membership problem, and we didn't want to reinvent the wheel for every system we built. Serf provided an incredibly stable and well proven foundation for us to build Consul on top, as well as future…

It would be pretty awesome if you could get some case studies out about these deployments of serf. I would really like to hear more about them.
Post reply on HN