Operator Framework: Building Apps on Kubernetes • Hacker News Board

Re: Operator Framework: Building Apps on Kubernetes

#21

Post by n0w » Tue, May 01, 2018, 10:07 PM UTC

cookiecaper wrote: post #5

> You may be familiar with Operators from the concept’s introduction in 2016. An Operator is a method of packaging, deploying and managing a Kubernetes application. "Operators", as introduced in 2016, were just bespoke Go programs that communicated with Kubernetes internals in a pretty low-level way. You were writing special-case plugins for Kubernetes, but they didn't want to make it sound that way, because I guess…

In a world where we are trying to codify more and more of the operational side of our applications, I personally look at operators as a better alternative to the configuration management systems we use to configure generic operating systems today.

We're moving away from an imperative configuration/operational model to a declarative one. While these operators target applicatings running in k8s I could imagine them being created to manage applications running elsewhere as well.

Re: Operator Framework: Building Apps on Kubernetes

#22

Post by jacques_chester » Tue, May 01, 2018, 10:29 PM UTC

thedevopsguy wrote: post #16

I've been an early adopter of docker. Used Compose when it was still called Fig, used and deployed kubernetes beta up to version 1 for in-house PAAS/heroku like environment. Must say I do miss those days when K8s was an idea that could fit in your head. The primitives were just enough back then. It was powerful developer tool for teams and we used it aggressively to accelerate our development process. K8s has now mov…

Operations is always more complex than folk expect it to be and product evolution typically reflects that. Kubernetes was simple because it couldn't miraculously teleport to do all the things ever-larger clusters require of it.

We forever rush to the limits of current technology and then blame the technology.

I think it's worth noting that Kubernetes never tried hard to impose an opinion about what belongs to the operator (as in the person running it) and what belongs to the developer. You get the box and then you work out amongst yourselves where to draw the value line.

Cloud Foundry, which came along earlier, took inspiration from Heroku and had a lot of folks of the convention-over-configuration school involved in its early days. The value line is explicitly drawn. It's the opinionated contract of `cf push` and the services API. That dev/ops contract allowed Cloud Foundry to evolve its container orchestration system through several generations of technology without developers having to know or care about the changes. From pre-Docker to post-Istio.

Disclosure: I work for Pivotal, we do Cloud Foundry stuff. But as it happens my current day job involves a lot of thinking about Kubernetes.

Re: Operator Framework: Building Apps on Kubernetes

#23

Post by dominotw » Tue, May 01, 2018, 10:29 PM UTC

thedevopsguy wrote: post #16

I've been an early adopter of docker. Used Compose when it was still called Fig, used and deployed kubernetes beta up to version 1 for in-house PAAS/heroku like environment. Must say I do miss those days when K8s was an idea that could fit in your head. The primitives were just enough back then. It was powerful developer tool for teams and we used it aggressively to accelerate our development process. K8s has now mov…

I just find docker swarm to be so much simpler and works for the type of stuff we deploy.

Sad to see it 'lose the race' against kubernetes.

Re: Operator Framework: Building Apps on Kubernetes

#24

Post by justinsaccount » Tue, May 01, 2018, 10:48 PM UTC

cookiecaper wrote: post #19

justinsaccount wrote: post #17

Earlier quoted context omitted.

> But we've had "rolling deployments" and "self-healing applications" before, without having to write 10k+ lines of code to manage the platform deployment You continue to be disingenuous and imply that every application requires 10k lines of code to run on k8s. I recently used k8s to deploy an application. To configure 2 services with exterior and interior load balancing and health checks for rolling deployments and…

> You continue to be disingenuous and imply that every application requires 10k lines of code to run on k8s. Let me clarify and state unambiguously that it won't necessarily take 10k lines of code to run any random application on Kubernetes. You can, in fact, deploy Prometheus without using the Prometheus operator and you'll technically be "running your monitoring" within k8s. It just isn't likely to be very reliable…

> If your existing setup was pretty tiny, this may have been a worthwhile project.

What existing setup? I wrote and deployed this application to k8s in the span of like 4 days. If I was using "real DNS, real haproxy, real nginx" I'd probably still be trying to work out how to do zero downtime rolling deployments, and then how to clone the whole thing so I could have a separate production environment.

Re: Operator Framework: Building Apps on Kubernetes

#25

Post by smarterclayton » Tue, May 01, 2018, 11:09 PM UTC

elsonrodriguez wrote: post #3

pm90 wrote: post #2

Can anyone talk about the positives/negatives of Operators v/s Helm Charts? From what I see, it seems like Operators are a better tool for defining and deploying Custom k8s resources whereas helm charts are good way to organize applications (deployment, service etc. templates packaged into one tar).

You can think of K8s Controllers/Operators as a way to extend the K8s API at runtime with a new type of object that is managed by a controller you've written yourself (as opposed to the built in k8s objects that are handled by the default controller-manager). A Helm chart by comparison is a way to template out K8s objects to make them configurable for different environments. Some shops will end up combining one with…

And that’s why the team created a Helm-operator based on Lostromos (a tool for expanding Helm charts without running a tillers erver) that makes it easy to use a Helm chart for templating but still add additional lifecycle on top:

https://github.com/operator-framework/helm-app-operator-kit

The templating aspect of Helm and the set of quality content is complementary to being able to add higher level lifecycle.

Re: Operator Framework: Building Apps on Kubernetes

#26

Post by cookiecaper » Tue, May 01, 2018, 11:12 PM UTC

justinsaccount wrote: post #24

cookiecaper wrote: post #19

Earlier quoted context omitted.

> You continue to be disingenuous and imply that every application requires 10k lines of code to run on k8s. Let me clarify and state unambiguously that it won't necessarily take 10k lines of code to run any random application on Kubernetes. You can, in fact, deploy Prometheus without using the Prometheus operator and you'll technically be "running your monitoring" within k8s. It just isn't likely to be very reliable…

> If your existing setup was pretty tiny, this may have been a worthwhile project. What existing setup? I wrote and deployed this application to k8s in the span of like 4 days. If I was using "real DNS, real haproxy, real nginx" I'd probably still be trying to work out how to do zero downtime rolling deployments, and then how to clone the whole thing so I could have a separate production environment.

Yeah, so there's the crux. If you're starting from scratch and you design something explicitly to fit within Kubernetes's constraints and demands, and those constraints and demands work well with the specific application you're designing, it will, of course, be a pleasant experience to deploy on the targeted platform. The same is true for anything else.

If you make your goal to "build something that runs great on Platform", it shouldn't be a surprise that the new thing you made runs great on it. I've been talking about Real Things That Already Exist and Run Real Businesses. That's usually what we're talking about when we talk about infrastructure and servers, and that's where we see this dangerous cargo culting where people don't realize "Just use an Operator" means "just write thousands of lines of highly-specific lifecycle management code so that Kubernetes knows how to do your things".

Re: Operator Framework: Building Apps on Kubernetes

#27

Post by ecnahc515 » Tue, May 01, 2018, 11:17 PM UTC

cookiecaper wrote: post #15

justinsaccount wrote: post #13

Earlier quoted context omitted.

> It appears there are over 20k lines of Go code after excluding vendor libraries and the example and contrib directories (arguably, contrib should've been included). close.. $ find -name '*.go'|grep -v ./vendor|xargs wc -l|sort -n|tail 507 ./test/e2e/alertmanager_test.go 540 ./pkg/client/monitoring/v1/types.go 562 ./pkg/alertmanager/operator.go 643 ./pkg/prometheus/statefulset.go 719 ./pkg/prometheus/promcfg.go 760…

> which is quite a bit off from 20k. True. I certainly admit that I don't know the code well enough to know exactly which lines or files are critical and which aren't, but I think it's getting into the weeds to nitpick the specifics too much (for example, I would argue that while it's probably good to exclude generated files, you shouldn't exclude the tests from the line count since they require real human time to ma…

You just said you don't understand the code well enough to discern which files matter and don't, then go to say it takes thousands of lines of code.

If you remove the generated code, and discount the tests, its barely a thousand. Much of it is the test code, and Go is fairly verbose for testing code.

It generally maybe takes a few hundred lines of code to write simple to moderately complex operators. A lot if it's generated and boilerplate. I would say a lot of that's due to lack of generics in Go, but I wouldn't say it's very much code overall. Additionally, the framework being presented here aims to reduce that down by removing the boilerplate, and making it easier to express the end goal (eg: self-healing, auto-rebalancing, etc) using less code.

It's certainly not much more code to implement an operator than what you would see in a well written Puppet module/Chef cookbook/Ansible playbook, and it does a lot more. You certainly could try to do self-healing using these tools, but it's significantly more difficult in my experience.

I agree that there's no free lunch and that you won't just necessarily get self-healing applications by using Kubernetes. But it's certainly easier to build them when using Kubernetes. The only thing that's really changed is instead of writing to a particular cloud provider API to handle this, you're able to leverage something more agnostic to the specific cloud/vendor you're using for your infrastructure.

Re: Operator Framework: Building Apps on Kubernetes

#28

Post by kkapelon » Tue, May 01, 2018, 11:34 PM UTC

merb wrote: post #20

thedevopsguy wrote: post #16

I've been an early adopter of docker. Used Compose when it was still called Fig, used and deployed kubernetes beta up to version 1 for in-house PAAS/heroku like environment. Must say I do miss those days when K8s was an idea that could fit in your head. The primitives were just enough back then. It was powerful developer tool for teams and we used it aggressively to accelerate our development process. K8s has now mov…

> You can see these operational patterns being used together to create a fairly advanced on-prem cloud infrastructure. At times, to me, it looks like over-engineering. well consider you wanted to have a High Available solutions that supports Blue Green / Rolling Deploys without downtime. You either built it yourself or you rely on something like k8s. It's not that much over engineering. K8s is a lot of code, yes. But…

>I think deploying k8s is still way easier than most other solutions out there, like all these PaaS's and Cloud solutions

Have you seen nomad + consul + traefik? Much easier to install and the end result is close to a K8s cluster.

Re: Operator Framework: Building Apps on Kubernetes

#29

Post by bryanlarsen » Tue, May 01, 2018, 11:50 PM UTC

nikolay wrote: post #12

mfer wrote: post #6

Earlier quoted context omitted.

Helm and Community Charts maintainer here... Helm is a package manager. Think of it like apt for Kubernetes. Operators enable you to manage the operation of applications within Kubernetes. They are complementary. You can deploy an operator as part of a Helm Chart. I recently wrote a blog post that explains how the different tools relate. https://codeengineered.com/blog/2018/kubernetes-helm-related...

Any plans to switch from templatized YAML to Jsonnet/Ksonnet?

The helm 3 proposal uses Lua. Not sure if they've actually started working on it: https://github.com/kubernetes-helm/community/blob/master/hel...

Re: Operator Framework: Building Apps on Kubernetes

#30

Post by justinsaccount » Tue, May 01, 2018, 11:56 PM UTC

cookiecaper wrote: post #26

justinsaccount wrote: post #24

Earlier quoted context omitted.

> If your existing setup was pretty tiny, this may have been a worthwhile project. What existing setup? I wrote and deployed this application to k8s in the span of like 4 days. If I was using "real DNS, real haproxy, real nginx" I'd probably still be trying to work out how to do zero downtime rolling deployments, and then how to clone the whole thing so I could have a separate production environment.

Yeah, so there's the crux. If you're starting from scratch and you design something explicitly to fit within Kubernetes's constraints and demands, and those constraints and demands work well with the specific application you're designing, it will, of course, be a pleasant experience to deploy on the targeted platform. The same is true for anything else. If you make your goal to "build something that runs great on Pla…

Well, that's not really what I did.

It was a variation of an earlier project that I had deployed to EC2.. just on EC2 I had a mess of fragile boto/fabric stuff to get the test/prod machines provisioned and the application installed. It "worked" but I had no redundancy and deploys were hard cut-overs.. and if the new version didn't come up for whatever reason, it was just down.

I didn't do anything in the application itself to design it to run on k8s, I was able to re-use some existing test code to define things like

        livenessProbe:
          exec:
            command:
            - /app/healthcheck.py
          initialDelaySeconds: 
          periodSeconds: 300
          timeoutSeconds: 5

so, 7 lines of yaml and I had self healing and rolling deployments. I could have built this out on EC2.. probably would have taken me a few hundred lines of terraform/ansible/whatever and never worked as well. It's the kind of thing where I could have maybe gotten rolling deployments working, but I would have just ended up with an "ad-hoc, informally-specified, bug-ridden, slow implementation of half of k8s"

I would have been perfectly happy to just run this whole thing on EB/heroku/lambda/whatever but the application was doing low level tcp socket stuff, not http, and almost every PaaS these days wants you to speak http.