Red Hat contributes etcd to the Cloud Native Computing Foundation • Hacker News Board

Re: Red Hat contributes etcd to the Cloud Native Computing Foundation

#31

Post by abronan » Sat, Dec 15, 2018, 1:56 PM UTC

bboreham wrote: post #22

philips wrote: post #4

etcd has succeeded as a piece of distributed systems infrastructure beyond our wildest expectation. When Alex Polvi, Xiang Li, and I started the project as a README in the summer of 2013 we identified that their still was no consensus database that was developer friendly, easily secured, production ready, and based on a well understood consensus algorithm. And largely we got lucky with good market timing, the inventi…

Docker Swarm Mode also embeds etcd. (The embedding mechanism is copy-paste, which I find both ingenious and a bit distasteful. Maybe I’m just sore I didn’t think of it first)

I wrote the initial implementation of the raft subsystem and it was definitely not a copy/paste. We started from scratch (using etcd's core raft) with the transport layer being grpc. My initial experiment could be found in this repository [1]. I then took the code from my initial experiment and included this into Swarmkit [2]. From there we went through many iterations on the initial code base and improved the UI with Docker swarm `init`/`join`/`leave` to make the experience of managing the cluster "friendly".

We spent quite some time evaluating different raft and paxos implementations (mostly Consul and etcd raft libraries), and found out etcd to be the most stable and flexible for our use case. It was very easy for example to swap the transport layer to use grpc. The fact that etcd implementation is represented as a simple state machine makes it also much easier to reason about under complex scenarios for debugging purposes, instead of digging into multiple layers of abstractions.

In retrospect, this came with quite a learning curve. We've had to deal with issues caused by our own misunderstandings on how to use the library properly. At the same time the fact that the developers favored stability as opposed to user friendliness was exactly what we found attractive using etcd's raft. Additionally, CoreOS developers were super friendly and helpful to help us fix these issues. We've reported and fixed some bugs as well. Kudos to them for all the help they provided at the time.

[1] https://github.com/abronan/proton [2] https://github.com/docker/swarmkit/commit/89de50f2092dfd2170...

Re: Red Hat contributes etcd to the Cloud Native Computing Foundation

#32

Post by crb » Sat, Dec 15, 2018, 2:41 PM UTC

RcouF1uZ4gsC wrote: post #6

I wonder if this the core of Redhat knowing that the clock is ticking, and make sure that critical software that they worked on is available in an open fashion. One only has to look at Sun and MySQL to see what can happen to a once vibrant open source offering after acquisition.

It was proposed in July (https://github.com/cncf/toc/issues/136) and voted on in September (https://github.com/cncf/toc/pull/143), all before the IBM/Red Hat deal was announced or known to the staff.

The announcement was held for publicity at KubeCon.

Re: Red Hat contributes etcd to the Cloud Native Computing Foundation

#33

Post by rbetts » Sat, Dec 15, 2018, 4:00 PM UTC

moosingin3space wrote: post #13

RcouF1uZ4gsC wrote: post #6

I wonder if this the core of Redhat knowing that the clock is ticking, and make sure that critical software that they worked on is available in an open fashion. One only has to look at Sun and MySQL to see what can happen to a once vibrant open source offering after acquisition.

Red Hat does not own anything that's valuable aside from their developers, who chose to work at Red Hat due to their pro-FOSS positioning. If IBM chose to start shutting projects like Fedora down or move in the direction of closed-source, these developers would have no desire to remain, and would leave, making that $40B acquisition worthless.

Red Hat has extensive customer relationships, customer databases, contracts for future revenue, partnerships, operational processes, . . . product IP is just a slice of the pie.

Re: Red Hat contributes etcd to the Cloud Native Computing Foundation

#34

Post by philips » Sat, Dec 15, 2018, 4:47 PM UTC

polskibus wrote: post #21

How does etcd compare to zookeeper? What made etcd the choice for kubernetes and cncf?

A few reasons we didn't use ZK at the time (some of these are out of date). Zookeeper had:

- No TLS security story

- An abandoned RPC/serialization system that was hard to use in other languages

- A consensus algorithm that differed from systems described in literature

- A large RAM footprint

Awhile ago some etcd engineers made an experiment in fact to try and run ZK client protocol on etcd with a proxy:

https://github.com/etcd-io/zetcd https://coreos.com/blog/introducing-zetcd

Today, etcd performs much better than ZK and I believe it is much more widely deployed with a wider set of engaged users.

https://coreos.com/blog/performance-of-etcd.html

Re: Red Hat contributes etcd to the Cloud Native Computing Foundation

#35

Post by bboreham » Sat, Dec 15, 2018, 4:51 PM UTC

abronan wrote: post #31

bboreham wrote: post #22

Earlier quoted context omitted.

Docker Swarm Mode also embeds etcd. (The embedding mechanism is copy-paste, which I find both ingenious and a bit distasteful. Maybe I’m just sore I didn’t think of it first)

I wrote the initial implementation of the raft subsystem and it was definitely not a copy/paste. We started from scratch (using etcd's core raft) with the transport layer being grpc. My initial experiment could be found in this repository [1]. I then took the code from my initial experiment and included this into Swarmkit [2]. From there we went through many iterations on the initial code base and improved the UI wit…

I apologise for my misunderstanding.

What I remember is, during DockerCon in June 2016, I went into the code to see how it worked, and I found a top-level file setting up data structures and handlers that seemed to be 90% the same as the equivalent file in etcd. And the underlying implementation is reused via vendoring.

Maybe this rings a bell with you and you can tell me what I saw, because I can't find it now.

Maybe I dreamed the whole thing.

I did, and still do, think integrating etcd into Swarm Mode was a masterstroke; we had spent the previous two years working to avoid "first you must install etcd" in a different way that nobody got. Afterwards we created kubeadm to ape the 'init' and 'join' functionality.

Re: Red Hat contributes etcd to the Cloud Native Computing Foundation

#36

Post by spullara » Sat, Dec 15, 2018, 6:06 PM UTC

pram wrote: post #19

spullara wrote: post #17

This is exactly the kind of thing that should be a layer on top of FoundationDB. Too many state stores all doing largely the same thing.

FoundationDB has only been open source for 8 months?? etcd is like 5 years old.

I meant going forward, not saying that etcd should have originally been built on top of it.

Re: Red Hat contributes etcd to the Cloud Native Computing Foundation

#37

Post by smarterclayton » Sat, Dec 15, 2018, 8:16 PM UTC

philips wrote: post #34

polskibus wrote: post #21

How does etcd compare to zookeeper? What made etcd the choice for kubernetes and cncf?

A few reasons we didn't use ZK at the time (some of these are out of date). Zookeeper had: - No TLS security story - An abandoned RPC/serialization system that was hard to use in other languages - A consensus algorithm that differed from systems described in literature - A large RAM footprint Awhile ago some etcd engineers made an experiment in fact to try and run ZK client protocol on etcd with a proxy: https://gith…

Pre 3.5.0 zookeeper reconfiguration of a running cluster was also much harder - that was a significant discussion point on Kubernetes when we had the etcd vs (anything) discussions early after open sourcing.

I still think etcd total ordering over history also made reasoning about changes in the system while we were writing the first versions of the controllers and caches and list-watch loops. ZK had partial order, and I was leery of that at the time.

Re: Red Hat contributes etcd to the Cloud Native Computing Foundation

#38

Post by dkoston » Mon, Dec 17, 2018, 4:26 PM UTC

polskibus wrote: post #29

guipsp wrote: post #25

Earlier quoted context omitted.

zookeeper uses a coordination kernel, while etcd uses a replicated state machine.

What are the pros and cons? Should etcd be the default whenever one is thinking about using something like that?

It really depends on your use case but one of the main "pros" of etcd is the narrow latency band when writing.

This article is likely biased to the good parts of etcd as it's written by coreOS but you can see how the latency of writes in etcd is very consistent compared to the wide range of latencies experienced writing to ZooKeeper or Consul:

https://coreos.com/blog/performance-of-etcd.html

There are other "pros" related to the fact that it's been designed for "cloud native" architectures like kubernetes. For example, FoundationDB can perform on average at sub-milisecond latency for writes (https://apple.github.io/foundationdb/benchmarking.html) versus 1.6ms on etcd however configuring FoudationDB to run programmatically is challenging as it was designed in an environment where ops people rack physical servers.

All key/value stores have good points and bad points but that's in relation to your use case. If write or read throughput isn't the most important, say it's consistency or availability, you may make a different choice about what are "pros" and what are "cons".

Another "pro" or "con" may be the language its written in or how it runs or deploys. If you run a Java shop and have tons of experience writing and deploying Java code, it may be in your best interest to be able to have more control by using a project written in Java. conversely, if you have all go engineers, you may want a project written in go. If you only have junior engineers, you may want whatever is easiest to operate and deploy.