Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Consul.io – Service discovery and configuration made easy

consul.io

Post reply on HN
Print view View original

21–30 of 35 posts

Re: Consul.io – Service discovery and configuration made easy

#21

Post by ch »

mitchellh wrote: post #9
bvanvugt wrote: post #6

Is anybody using this in large production systems?

(Note: I'm one of the folks who works at HashiCorp) We don't have permission at this time to share some of our big users (we're working on approvals!), but I can say that that some very big companies you've certainly heard of and probably use right now have deployed Consul across every server. And Consuls deployment in just companies were working with is in the hundreds of thousands of machines. At this point were ve…

When you say hundreds of thousands of machines, is each one running a consul agent, or do some just make use of answers provided via DNS or the remote api?

Without divilging too many details, what would be the ratio of agents running '-server' to overall agents?

Re: Consul.io – Service discovery and configuration made easy

#22

Post by jimktrains2 »

armon wrote: post #15
bascule wrote: post #11

Consul looks awesome. The only thing I don't like about it is they chose to use a homebrew encryption scheme for Serf's gossip protocol: http://www.serfdom.io/docs/internals/security.html (If you're confused about how this relates to Consul, see this: http://www.consul.io/docs/internals/security.html ) Worse, their justification for using this instead of standard schemes like (D)TLS seems to be that they don't need t…

We did take a look at D(TLS) when implementing Serf and the associated encryption mechanism. D(TLS) is much better suited when you are doing point-to-point duplex communication. The gossip algorithm instead is doing peer-to-peer (N-to-N instead of 1-to-1) in a half-duplex model. Using DTLS would've been too heavy for our use case. We did develop the algorithm in the open (see this gist: https://gist.github.com/armon/…

> The gossip algorithm instead is doing peer-to-peer (N-to-N instead of 1-to-1) in a half-duplex model.

Unless you mean you're doing multicast, you're still doing 1-to-1 connections.

> We did develop the algorithm in the open and made a call on the community to provide feedback to help improve the design of the system.

I wouldn't trust myself, let alone many unknown people who are just volunteering their time and have no known credentials as a cryptographer.

> instead stuck to the best practices around the most modern systems.

Except for the whole "don't role your own crypto system" one.

There are just so many things that can go wrong and when crypto fails it can do so very quietly. Additionally, if you ever want anyone to interoperate with you it'll just be a PITA for them, and depending on who it is, a PITA for you.

The overhead of a proper protocol like (D)TLS isn't that much.

> "On our production frontend machines, SSL/TLS accounts for less than 1% of the CPU load, less than 10 KB of memory per connection and less than 2% of network overhead. Many people believe that SSL/TLS takes a lot of CPU time and we hope the preceding numbers will help to dispel that." - Adam Langley, Google

Re: Consul.io – Service discovery and configuration made easy

#23

Post by armon »

jimktrains2 wrote: post #22
armon wrote: post #15

Earlier quoted context omitted.

We did take a look at D(TLS) when implementing Serf and the associated encryption mechanism. D(TLS) is much better suited when you are doing point-to-point duplex communication. The gossip algorithm instead is doing peer-to-peer (N-to-N instead of 1-to-1) in a half-duplex model. Using DTLS would've been too heavy for our use case. We did develop the algorithm in the open (see this gist: https://gist.github.com/armon/…

> The gossip algorithm instead is doing peer-to-peer (N-to-N instead of 1-to-1) in a half-duplex model. Unless you mean you're doing multicast, you're still doing 1-to-1 connections. > We did develop the algorithm in the open and made a call on the community to provide feedback to help improve the design of the system. I wouldn't trust myself, let alone many unknown people who are just volunteering their time and hav…

You are right, the communication is still unicast in nature. I should clarify to say that there isn't a persistent 1-to-1 communication, the nodes we gossip with are randomly selected on each interval. There is no connection or session establishment between peers.

I guess it depends on your definition of roll your own. We didn't invent AES-GCM or implement it. We are using the implementation shipped with the Golang stdlib.

Re: Consul.io – Service discovery and configuration made easy

#24

Post by sandstrom »

ch wrote: post #21
mitchellh wrote: post #9

Earlier quoted context omitted.

(Note: I'm one of the folks who works at HashiCorp) We don't have permission at this time to share some of our big users (we're working on approvals!), but I can say that that some very big companies you've certainly heard of and probably use right now have deployed Consul across every server. And Consuls deployment in just companies were working with is in the hundreds of thousands of machines. At this point were ve…

When you say hundreds of thousands of machines, is each one running a consul agent, or do some just make use of answers provided via DNS or the remote api? Without divilging too many details, what would be the ratio of agents running '-server' to overall agents?

I'm no expert on consul, but I think it's architected such that you'll run a few servers (say 3 or 5) and the rest will be agents.

All machines using consul will (generally) run the agent locally.

Re: Consul.io – Service discovery and configuration made easy

#25

Post by bascule »

armon wrote: post #23
jimktrains2 wrote: post #22

Earlier quoted context omitted.

> The gossip algorithm instead is doing peer-to-peer (N-to-N instead of 1-to-1) in a half-duplex model. Unless you mean you're doing multicast, you're still doing 1-to-1 connections. > We did develop the algorithm in the open and made a call on the community to provide feedback to help improve the design of the system. I wouldn't trust myself, let alone many unknown people who are just volunteering their time and hav…

You are right, the communication is still unicast in nature. I should clarify to say that there isn't a persistent 1-to-1 communication, the nodes we gossip with are randomly selected on each interval. There is no connection or session establishment between peers. I guess it depends on your definition of roll your own. We didn't invent AES-GCM or implement it. We are using the implementation shipped with the Golang s…

Nobody is questioning whether AES-GCM is a good algorithm or not, however you are using AES-GCM as part of a hand-rolled transport encryption protocol, and this is what's worrisome. Designing a transport encryption protocol is a difficult endeavor, and it seems you have skipped most of the steps (e.g. replay attack prevention) but suggest that it's irrelevant because other parts of the protocol provide security (e.g. the SWIM state machine).

This makes your protocol difficult to audit: someone concerned about potential attacks can't just look at your protocol in isolation, but has to factor the underlying protocol state machine into the security of your transport encryption protocol.

Re: Consul.io – Service discovery and configuration made easy

#26

Post by raspasov »

Love the nice informative website.

However:

[1] https://www.consul.io/intro/vs/zookeeper.html

This did not unequivocally convince me in the benefits over ZooKeeper. On the contrary, it makes it seems that ZooKeeper tries to do less, and I strongly prefer simpler tools.

[2] http://aphyr.com/posts/316-call-me-maybe-etcd-and-consul

Have the issues discussed in this article been fully addressed yet?

Overall, can you outline a use case where Consul is definitely better than ZooKeeper?

Re: Consul.io – Service discovery and configuration made easy

#27

Post by ch »

sandstrom wrote: post #24
ch wrote: post #21

Earlier quoted context omitted.

When you say hundreds of thousands of machines, is each one running a consul agent, or do some just make use of answers provided via DNS or the remote api? Without divilging too many details, what would be the ratio of agents running '-server' to overall agents?

I'm no expert on consul, but I think it's architected such that you'll run a few servers (say 3 or 5) and the rest will be agents. All machines using consul will (generally) run the agent locally.

You have the general topology right, but not all nodes necessarily have to be agents. You could, forinstance, run an agent next to a bind server and delegate the .consul. Zone to it. That way for nodes which only use DNS for discovery can work I modified.

In general you are going to set up an N+M topology of n servers and n+m agents -- but for a network of 100k nodes does consul scale to 100k agents? Plus what goes unstated is how those 100k nodes are laid out in DCs and racks.

Anyway, this isn't to slag on consul. I think consul is like chocolate and peanut butter.

I use it in my mesos cluster and just am genuinely curious how large it can scale!

Re: Consul.io – Service discovery and configuration made easy

#28

Post by armon »

raspasov wrote: post #26

Love the nice informative website. However: [1] https://www.consul.io/intro/vs/zookeeper.html This did not unequivocally convince me in the benefits over ZooKeeper. On the contrary, it makes it seems that ZooKeeper tries to do less, and I strongly prefer simpler tools. [2] http://aphyr.com/posts/316-call-me-maybe-etcd-and-consul Have the issues discussed in this article been fully addressed yet? Overall, can you outl…

The issues discussed by aphyr have been fully fixed as of Consul 0.2, several months ago. In the initial release of Consul, only the "default" mode was available and has the now-documented caveat of a stale read possibility. The various consistency modes are now discussed here: https://www.consul.io/docs/internals/consensus.html#default

With respect to ZooKeeper, there are different approaches. ZK provides a low-level primitive on which you can build. Consul provides similar primitives, but it ships with many features out of the box that don't require any development effort. It's a "batteries included" approach.

Specific examples:

- Real-time configuration with Consul + consul-template

- DNS based service discovery

- Scalable Nagios replacement

- Dynamic HAProxy / Varnish configuration

- Application configuration with Consul + envconsul

- Triggering config management tools with the event system

That is just a handful of uses for Consul that don't require writing any code. Doing similar things with ZK is possible, just requires a lot more work.

Re: Consul.io – Service discovery and configuration made easy

#29

Post by nhjk »

krat0sprakhar wrote: post #8

For people interested in Distributed Systems the architecture[0] page is a treasure trove. The explanation of the gossip(SWIM) and consensus protocol(Raft) used is easy to understand and quite lucid. It would be wonderful if all (most?) open-source projects have a such beautiful documentation explaining their architecture. Thank you Armon and team for sharing this with us! [0] - https://consul.io/docs/internals/archi…

Any suggestions on where to get an in depth intro to distributed systems in general?
Post reply on HN