The etcd operator: Simplify etcd cluster configuration and management • Hacker News Board

Re: The etcd operator: Simplify etcd cluster configuration and management

#31

Post by philips » Fri, Nov 04, 2016, 2:48 AM UTC

ex3ndr wrote: post #29

Can someone clarify some points? * Isn't etcd2 is required to start kubernetes? I found that if etcd2 is not helaty or connection is just temporary lost then k8s just freezes it's scheduling and API. So what if Operator and etcd2 is working on one node and it is down? Also i found that etcd2 also freezes event when one node is down. Isn't it unrecoverable situation? * k8s/coreos manual recommends to have etcd2 server…

1) Yes, Kubernetes relies on etcd as its primary database. Right now the etcd Operator does not tackle trying to manage the etcd that Kubernetes relies on. But! We are working on that as part of our self-hosted work https://coreos.com/blog/self-hosted-kubernetes.html. Stay tuned.

2) etcd can deal with any latency up to seconds long for say a globally replicated etcd. But! You need to tune etcd to expect that latency so it doesn't trigger a leader election. See the tuning guide: https://coreos.com/etcd/docs/latest/tuning.html

3) The backups are something that we are just getting to with the etcd Operator. Our intention is to help you create backups and create new clusters from arbitrarily old backups, but that work hasn't started yet.

Re: The etcd operator: Simplify etcd cluster configuration and management

#32

Post by theptip » Fri, Nov 04, 2016, 3:03 AM UTC

russell_h wrote: post #2

I've been thinking about implementing a custom controller that would use Third Party Resources as a way to install and manage an application on top of Kubernetes. The way that Kuberetes controllers work (watching a declarative configuration, and "making it so") seems like a great fit for the problem. Its exciting to see CoreOS working in the same direction - this looks much more elegant than what I would have hacked…

I've been thinking the same way; the k8s Third Party Resource API really enables some clever solutions.

While most k8s users are (from what I can tell) currently writing YAML config files and loading them by hand (encouraged by tools like Helm and Spread), I think that the k8s apps of the future will be more like the Operator;

1) The 'deploy scripts' are controllers that run in your k8s cluster and dynamically ensure the rest of your code is running, and the primitives that you operate on will be your custom ThirdPartyResources.

2) All of the config for your app is wrapped in a domain-specific k8s object spec; instead of writing a YAML file and uploading it as a raw Deployment, you would create a FooService API object with just the parameters that you actually care about for configuring your service.

Right now it's a pain and a lot of code (>10kloc of Go for the etcd-operator!) but I'm sure that a bunch of that could be abstracted out into a framework that makes it easy to generate/build operators for a variety of application use-cases.

Currently the solutions that build and deploy your code for you in k8s seem to be PaaS replacements (Deis, Openshift), which take a very generic approach to bundling your code. That's probably going to work for common use-cases, but I suspect the more bespoke deployments will need something more like the Operator approach, and I'm looking forward to seeing what tooling evolves in this area.

Re: The etcd operator: Simplify etcd cluster configuration and management

#33

Post by SEJeff » Fri, Nov 04, 2016, 3:38 AM UTC

philips wrote: post #10

theptip wrote: post #7

Earlier quoted context omitted.

The FAQ at the end of the OP addresses this: "Q: How is this different than StatefulSets (previously PetSets)? A: StatefulSets are designed to enable support in Kubernetes for applications that require the cluster to give them "stateful resources" like static IPs and storage. Applications that need this more stateful deployment model still need Operator automation to alert and act on failure, backup, or reconfigure.…

Darren, the FAQ is at the overview post here: https://coreos.com/blog/introducing-operators.html

Do you ever see this pattern being used to host/run the etcd cluster for the k8s apiserver to speak with? That would be quite meta (and kind of amazing if it would actually work).

Re: The etcd operator: Simplify etcd cluster configuration and management

#34

Post by philips » Fri, Nov 04, 2016, 3:56 AM UTC

SEJeff wrote: post #33

philips wrote: post #10

Earlier quoted context omitted.

Darren, the FAQ is at the overview post here: https://coreos.com/blog/introducing-operators.html

Do you ever see this pattern being used to host/run the etcd cluster for the k8s apiserver to speak with? That would be quite meta (and kind of amazing if it would actually work).

This is in progress! We have it prototyped out in bootkube. This is the plan. :)

Re: The etcd operator: Simplify etcd cluster configuration and management

#35

Post by SEJeff » Fri, Nov 04, 2016, 4:06 AM UTC

philips wrote: post #34

SEJeff wrote: post #33

Earlier quoted context omitted.

Do you ever see this pattern being used to host/run the etcd cluster for the k8s apiserver to speak with? That would be quite meta (and kind of amazing if it would actually work).

This is in progress! We have it prototyped out in bootkube. This is the plan. :)

Oh this is absolutely brilliant! So with the kubelet + systemd, you more or less have self managed everything. Shutup and take my money :)

Re: The etcd operator: Simplify etcd cluster configuration and management

#36

Post by hosh » Fri, Nov 04, 2016, 5:14 PM UTC

philips wrote: post #19

hosh wrote: post #14

This is brilliant. It's like the promise-theory-based convergence tools (CFEngine, Puppet, Chef) on top of K8S primitives. Better yet, the extension works like other K8S addons -- you start it up by scheduling the controller pod. That means potentially, I could use it in say, GKE, which I might not have direct control over the kube-master. I wonder if it is leverging PetSets. I also wonder how this overlaps or plays…

All of your questions are answered in the FAQ section of the overview post: https://coreos.com/blog/introducing-operators.html

I don't think my specific questions are answered by FAQ on that page.

The only answer I found that addresses one part of what I'm wondering about is "How is this different from configuration management like Puppet or Chef?" However, I did not ask that question.

If you read some of the Mike Burgess's "Promise Theory: Principles and Applications", you'll realize that Operators (and Kubernetes controllers for that matter) are applications and implementations of specific parts of the Promise Theory. This idea that Puppet or Chef is "configuration management" is a story sold to non-technical people. I would argue that Operators may be a _better_ application of Promise Theory than previous-generation tools.

Puppet or Chef running as a Kubernetes controller might be able to twiddle things. It's not exactly a great fit because both would be calling each respective servers rather than using Third Party Resources on the kube master (and such, unwieldy) The DSL in each would have to be extended for things useful for controlling a Kubernetes cluster, but once in place, it can do exactly what that etcd Operator does: converge on the desired state by managing memberships and doing cleanups.

Don't get me wrong: I like the CoreOS technology as well as Kubernetes. I've deployed on CoreOS and Kubernetes before. I get that companies have a responsibility to control the story and the messaging ... but what I am asking are questions that are bigger than any single technology or company, and I like to make up my own mind about things.