Consul, a new tool for service discovery and configuration

I'll preface these comments by saying that Consul appears to be the first distributed cluster management tool i've seen in years that gets pretty much everything right (I can't tell exactly what their consistency guarantees are; I suppose it depends on the use case?).

What I will say, in my usually derisive fashion, is I can't tell why the majority of businesses would need decentralized network services like this. If you own your network, and you own all the resources in your network, and you control how they operate, I can't think of a good reason you would need services like this, other than a generalized want for dynamic scaling of a service provider (which doesn't really work without your application being designed for it, or an intermediary/backend application designed for it).

Load balancing an increase of requests by incrementally adding resources is what most people want when they say they want to scale. You don't need decentralized services to provide this. What do decentralized services provide, then? "Resilience". In the face of a random failure of a node or service, another one can take its place. Which is also accomplished with either network or application central load balancing. What you don't get [inherently] from decentralized services is load balancing; sending new requests to some poor additional peer simply swamps it. To distribute the load amongst all the available nodes, now you need a DHT or similar, and take a slight penalty from the efficiency of the algorithm's misses/hits.

All the features that tools like this provide - a replicated key/value store, health checks, auto discovery, network event triggers, service discovery, etc - can all be found in tools that work based on centralized services, while remaining scalable. I guess my point is, before you run off to your boss waving an iPad with Consul's website on it demanding to implement this new technology, try to see if you need it, or if you just think it's really cool.

It's also kind of scary that the ability of an entire network like Consul's to function depends on minimum numbers of nodes, quorums, leaders, etc. If you believe the claims that the distributed network is inherently more robust than a centralized one, you might not build it with fault-tolerant hardware or monitor them adequately, resulting in a wild goose chase where you try to determine if your app failures are due to the app server, the network, or one piece of hardware that the network is randomly hopping between. Could a bad switch port cause a leader to provide false consensus in the network? Could the writes on one node basically never propagate to its peers due to similar issues? How could you tell where the failure was if no health checks show red flags? And is there logging of the inconsistent data/states?

See also SkyDNS, another service discovery system: http://blog.gopheracademy.com/skydns

In common with Consul:

* DNS interface

* Operates as a distributed cluster

* Uses Raft for consensus

Looks like a very cool tool -- could replace Zookeeper with saner admin requirements -- but I'm more interested in the tech. AP systems (such as Serf, on which Consul is built) have many advantages and I think we're only just beginning to see their adoption. I believe CRDTs are the missing ingredient to restore sanity to inconsistent data. Add that and I can see a lot more such systems being deployed in the future (and particularly in mine :-)

Oh, well i'm a giant ass then. I saw "completely distributed" and extrapolated decentralized from that. Still, an automatically-[randomly?]-elected leader in a distributed network is a form of decentralized network. It's effectively a toroidal network topology, since your datacenters make it multi-dimensional (though now that I look at hypercubes i'm not exactly sure which is a better fit here)

Now that i've re-read your architecture page, let me see if I understand this: the basic point behind using Consul is to have multiple servers agree on the result of a request, and communicate that agreement to a single node to write it, and then return it to the client. So really it's a fault-tolerant messaging platform that includes features that take advantage of such a network; do I have that right?

Also, your docs say there are between three and five servers, but here you're saying you only need one?