This is really awesome, distributed system techniques in the real world. I'm really jealous of what they've managed to build. I was planning to make a tool like this (smaller scale, one machine), and this will certainly serve as a good guide on how to do it right (or whether I should even bother at all). I can't find a trace of a standard/included slick web interface for managing the clusters and agents -- are they l…
Consul, a new tool for service discovery and configuration
21–30 of 62 posts
Re: Consul, a new tool for service discovery and configuration
#22This is really awesome, distributed system techniques in the real world. I'm really jealous of what they've managed to build. I was planning to make a tool like this (smaller scale, one machine), and this will certainly serve as a good guide on how to do it right (or whether I should even bother at all). I can't find a trace of a standard/included slick web interface for managing the clusters and agents -- are they l…
Re: Consul, a new tool for service discovery and configuration
#23Should something be happening with the bar data payload in the HTTP kv example? Or is the value encoded for some reason?
Re: Consul, a new tool for service discovery and configuration
#24Registered services and nodes can be queried using both a DNS interface as well as an HTTP interface. This is very cool. Integrating with a name resolution protocol that every existing programmer and stack knows how to use (often without even thinking about it) should lead to some magical "just works" moments.
Re: Consul, a new tool for service discovery and configuration
#25How much time did it take to put this together?
Re: Consul, a new tool for service discovery and configuration
#26This is really awesome, distributed system techniques in the real world. I'm really jealous of what they've managed to build. I was planning to make a tool like this (smaller scale, one machine), and this will certainly serve as a good guide on how to do it right (or whether I should even bother at all). I can't find a trace of a standard/included slick web interface for managing the clusters and agents -- are they l…
We didn't manage to finish it in time, but it should be released in the next two weeks or so!
If I may ask, it seems like the design of the consul site is one step (iteration) away from the serf site (particularly, the docs pages -- some subtle changes made a large difference)... I agree with the others here, really dig the site, and big text definitely doesn't hurt deeply technical descriptions architecture page was very readable for me
Re: Consul, a new tool for service discovery and configuration
#27Earlier quoted context omitted.
We didn't manage to finish it in time, but it should be released in the next two weeks or so!
I can only imagine how awesome it's going to be, given the excellent design (simple, readable, content-focused) of the other work you guys are doing. If I may ask, it seems like the design of the consul site is one step (iteration) away from the serf site (particularly, the docs pages -- some subtle changes made a large difference)... I agree with the others here, really dig the site, and big text definitely doesn't…
Re: Consul, a new tool for service discovery and configuration
#28How does this differ from http://www.serfdom.io/ , another HashiCorp product?
Re: Consul, a new tool for service discovery and configuration
#29Earlier quoted context omitted.
So I think here the problem isn't really the datastore, it's more the high-availability and discovery. The main bonus that Consul seems to be providing is maintaining a logical topology of your network without you doing much. They do this by a using gossip-based protocol and a derivative of paxos called Raft. These two things work together to essentially have the servers that run your various services (whether api or…
Wouldn't a centralized Cassandra cluster be reliable enough to meet that need? I've never had a cluster completely collapse on me unless things were already screwed up enough that Service Discovery was ultimately useless since nothing else would work. It just seems to me that losing your datastore makes your services unusable...at which point 'discovering them' isn't really the issue. Instead, everyone wants to intro…
Well so the problem is, as soon as you centralize, you introduce a single point of failure, which is a no-no if you're looking for as pure of a distributed system as you can get (distributed systems have their flaws, but single-point-of-failure systems have been worked past at this point, generally the drawbacks are expressed in terms of number of non-faulty/byzantine nodes).
While it is definitely true that if the cluster completely collapses, service discovery won't work anyway, but as that is very rare (hopefully), the thinking here is that what if your centralized cassandra cluster fails? You would need to replicate everything to something else, and once you start preparing for those kinds of failures, you're already building a distributed system.
NOTE: I am assuming here that you mean ONE machine running cassandra... if you mean multiple, then the stuff below doesnt' really apply, if cassandra handles dynamic node changing well... but still, why not abstract? Why not make EVERY service you're running app/db/cache/app2/utility know about dynamic changes to architecture?
What do you mean by "losing your data store?" -- from what I understand, a consul agent runs on every machine and EVERY consul agent has an LMDB instance. If you mean losing your data store as in losing the service that provides your actual application data -- that would be the point of automatically discovering services, you could just arbitrarily add nodes that do the "db" service, and your nodes that run "app" would automatically know more "db"s showed up.
Forgive me if this is unnecessary explanation, but:
To illustrate this -- let's say I have 3 servers, 2 are running instances of the app (5 instances each) and 1 big-RAM machine is running the DB. All 10 instances are relying on that DB to not go down. While there are many very very capable & reliable DBs out there (cassandra, postgres, etc), it's dangerous to assume they will not fail.
However, the problem is, how do you just add nodes? You're going to either need to change app code, change some env variables, or do some other kind of monkey patching to let some of the app processes (there are 10 of them) know which DB to use. Also, if you look at just the problem of adding instances of app processes for more load balancing, there are various static-y files that possibly need to change to accomodate (nginx/apache config, env variables,etc).
Again, someone correct me if I'm wrong, but this is where Consul comes in. If app server 1 knows about at LEAST 1 of the DB clusters, you can easily add more DB clusters, and ask Consul about them. So, if one DB has gone down, and you have consul-aware code in place, consul can tell your app instances where to get their database data.
Like Cassandra, There are some DBs that make this really easy to do (spin up more DBs that can act as masters, or just backup read onlys, or whatever) -- rethinkdb is one of them (http://rethinkdb.com/)... They have a really good web interface that makes adding and managing clusters as easy as starting up a rethinkdb service with some extra options telling it where the master is. However, cassandra seems like it doesn't really handle dynamic node creation (I'm going off this page: http://www.datastax.com/docs/0.8/install/cluster_init). If it does, the case for an abstracted, dynamic service discovery still stands (cassandra might be OK, what about if you want to know about service x?)
Re: Consul, a new tool for service discovery and configuration
#30Earlier quoted context omitted.
Wouldn't a centralized Cassandra cluster be reliable enough to meet that need? I've never had a cluster completely collapse on me unless things were already screwed up enough that Service Discovery was ultimately useless since nothing else would work. It just seems to me that losing your datastore makes your services unusable...at which point 'discovering them' isn't really the issue. Instead, everyone wants to intro…
tldr; centralize, and you have single point of failure, that's bad for distributed systems. you can have multiple DBs, but how will your app know about them, if they're brought up dynamically? You'll have to stop instances, possibly modify some code/env/something, and keep those changes in your head. Well so the problem is, as soon as you centralize, you introduce a single point of failure, which is a no-no if you're…
I think you misunderstood what I was talking about based on your explanation with a single physical machine running a single database instance and manually adding nodes requiring human intervention.