Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The etcd operator: Simplify etcd cluster configuration and management

coreos.com

Post reply on HN
Print view View original

21–30 of 36 posts

Re: The etcd operator: Simplify etcd cluster configuration and management

#21

Post by invisible »

darren0 wrote: post #13
philips wrote: post #10

Earlier quoted context omitted.

Darren, the FAQ is at the overview post here: https://coreos.com/blog/introducing-operators.html

Thanks, but this alludes to more operators coming. Why is such a thing really needed. The proliferation of this approach seems like a potential downfall of k8s. Similar to Mesos where a framework is quite powerful, the cost of developing one is too high. This blog post basically implies that k8s base constructs can't run postgres, redis, prometheus, etcd, cassandra, etc. But why? Are we saying that stateful services…

The reason is that while replication controllers and daemonsets fulfill making sure there are enough of xyz pod, they do not ensure that replication is set up between those pods. For example, in redis one must do `slaveof` for slaves if implementing basic replication. If the master goes down, a new master must be appointed and then `slaveof` configured on afterward. That in itself requires either a) an operator or b) concensus among enough nodes to reach a quorom about these.

So while kubernetes could be the authority (via the master), generally it's only meant for scheduling pods. There would have to be some better guarantees about master availability if it were to control these things.

Re: The etcd operator: Simplify etcd cluster configuration and management

#22

Post by tristanz »

philips wrote: post #18
darren0 wrote: post #13

Earlier quoted context omitted.

Thanks, but this alludes to more operators coming. Why is such a thing really needed. The proliferation of this approach seems like a potential downfall of k8s. Similar to Mesos where a framework is quite powerful, the cost of developing one is too high. This blog post basically implies that k8s base constructs can't run postgres, redis, prometheus, etcd, cassandra, etc. But why? Are we saying that stateful services…

I don't think stateful applications require the use of something like an Operator. It really just comes down to where the state lives. For example if you are wanting to run your database on top of an EBS, SAN, or something like that it is no problems to just throw up a StatefulSet and go for it. However, if you start to think about things like orchestrating scaling of databases that have a administrative tool like Ca…

I think philips is exactly right. The whole design of Kubernetes is geared toward allowing users write their own controllers for advanced use cases. You could view something like Jenkins or Vitess as controllers because they spawn Kubernetes pods on demand. The beauty is Kubernetes gives you great primitives, so you often will be controlling these objects, not the underlying pods. Of course many simple applications don't need a controller, although I suspect more and more simple use cases will be managed by an external controller like Helm which orchestrates the lifecycle of applications.

The questions is: Why isn't this just called a controller? What's this new term Operator?

Re: The etcd operator: Simplify etcd cluster configuration and management

#23

Post by doublerebel »

0x74696d wrote: post #15
doublerebel wrote: post #8

Earlier quoted context omitted.

Thanks, I remember seeing the autopilot pattern mentioned on Joyent's blog, but haven't seen that website. The lifecycle [0] looks remarkably similar to the build and deployment steps outlined in Distelli's manifest [1]. I use Distelli+Consul on Joyent so I suppose I've been doing the autopilot pattern without realizing it! I know that much of Distelli's workflow comes from the founders' experience at AWS, so I wonde…

I'm the lead developer for Joyent of ContainerPilot, which is the tool at the core of our Autopilot Pattern implementation examples. The lifecycle events you recognize in Distelli are definitely similar. And Chef's new tool Habitat has a supervisor that was independently developed but ended up having interesting parallels with ContainerPilot. So there's a universal idea lurking under there, which is why we called Aut…

Thanks for sharing those details, great to have more insight into the process.

In my case the Distelli agent does live inside the "container", because I'm using SmartOS instances and not Docker containers. It handles deployment, and monitors processes of the apps when I'm not using an SMF.

I'm not sure how Distelli's K8s orchestration works, that functionality is more recent. In my case, the lifecycle details are in the manifest in the app repo, which is just a YAML where each lifecycle section is a bash script. App builds are just tarballs in S3. So there's not much to the deployment process.

Re: The etcd operator: Simplify etcd cluster configuration and management

#25

Post by ideal0227 »

hatred wrote: post #11

The concept of custom controllers looks similar to what schedulers are in Mesos. It's nice to see the two communities taking a leaf out of each other's books e.g., Mesos would introduce experimental support for task groups (aka Pods) in 1.1. Disclaimer: I work at Mesosphere on Mesos.

Yea. They are similar in functionality.

But they work differently. The operator does not really “schedule” containers. It finishes the controlling logic by using Kubernetes APIs. For example, it uses native Kubernetes health checking, service discovery, deployment. It works completely on top of Kubernetes API, so no specialized scheduler, executor or proxy are needed comparing to https://github.com/mesosphere/etcd-mesos/blob/master/docs/ar....

The advantages of Mesos is exposing lower level APIs and resources to allow more control. The etcd operator we built does not really need that. Building this kind of application operator may be simpler on k8s than on native Mesos.

Disclaimer: I work at CoreOS on Kubernetes and etcd.

Re: The etcd operator: Simplify etcd cluster configuration and management

#26

Post by Terretta »

philips wrote: post #10
theptip wrote: post #7

Earlier quoted context omitted.

The FAQ at the end of the OP addresses this: "Q: How is this different than StatefulSets (previously PetSets)? A: StatefulSets are designed to enable support in Kubernetes for applications that require the cluster to give them "stateful resources" like static IPs and storage. Applications that need this more stateful deployment model still need Operator automation to alert and act on failure, backup, or reconfigure.…

Darren, the FAQ is at the overview post here: https://coreos.com/blog/introducing-operators.html

"An Operator represents human operational knowledge in software, to reliably manage an application."

Loving this.

Re: The etcd operator: Simplify etcd cluster configuration and management

#27

Post by why-el »

Somewhat unrelated, but I am just curious. For those who use etcd (and this is coming from a place of ignorance), does the key layout (which the keys are currently stored, how they are structured) get out of hand? Meaning, does it get to a place where a dev working with etcd might not have an idea about what in etcd at any given time? Or do teams force some kind of policy (in documentation or code) that everyone must respect?

I am asking because I was in situation where I was introduced to other key-value stores, and because the team working with them is big and no process was followed to group all keys in one place, it was hard to know "what is in the store" at any moment, short of exhausting all the entry points in the code.

Re: The etcd operator: Simplify etcd cluster configuration and management

#28

Post by jbpetersen »

duaneb wrote: post #12
jbpetersen wrote: post #4

Being someone who's been getting more familiar lately with backend engineering and has been trying to make sense of various options, I've got a strong enough impression of CoreOS that I'm betting my time it'll be dominating the next few years. I also can't wait to see an open version of AWS Lambda / Google Functions appear.

There are already lambda implementations available; I can't speak to google functions.

Is there a significant difference?

Re: The etcd operator: Simplify etcd cluster configuration and management

#29

Post by ex3ndr »

Can someone clarify some points?

* Isn't etcd2 is required to start kubernetes? I found that if etcd2 is not helaty or connection is just temporary lost then k8s just freezes it's scheduling and API. So what if Operator and etcd2 is working on one node and it is down? Also i found that etcd2 also freezes event when one node is down. Isn't it unrecoverable situation?

* k8s/coreos manual recommends to have etcd2 servers not that far from each other mostly because it have very strict requirements about networks (ping 5ms or so) that for some pairs of servers couldn't work well.

* What if we will lost ALL nodes and it will create almost new cluster from backups, but what if we will need to restore latest version (not 30 mins ago)?

Re: The etcd operator: Simplify etcd cluster configuration and management

#30

Post by philips »

tristanz wrote: post #22
philips wrote: post #18

Earlier quoted context omitted.

I don't think stateful applications require the use of something like an Operator. It really just comes down to where the state lives. For example if you are wanting to run your database on top of an EBS, SAN, or something like that it is no problems to just throw up a StatefulSet and go for it. However, if you start to think about things like orchestrating scaling of databases that have a administrative tool like Ca…

I think philips is exactly right. The whole design of Kubernetes is geared toward allowing users write their own controllers for advanced use cases. You could view something like Jenkins or Vitess as controllers because they spawn Kubernetes pods on demand. The beauty is Kubernetes gives you great primitives, so you often will be controlling these objects, not the underlying pods. Of course many simple applications d…

Controller wasn't quite the term to capture the combination of an application specific controller and third party resource to manage a collection of user created application instances.

So, we arrived at Operator. It felt like a good term that we could put after X that encapsulates the intent of the pattern. It helps you operate instances of an application.

Post reply on HN