Operator Framework: Building Apps on Kubernetes

close..

  $ find -name '*.go'|grep -v ./vendor|xargs wc -l|sort -n|tail
     507 ./test/e2e/alertmanager_test.go
     540 ./pkg/client/monitoring/v1/types.go
     562 ./pkg/alertmanager/operator.go
     643 ./pkg/prometheus/statefulset.go
     719 ./pkg/prometheus/promcfg.go
     760 ./test/e2e/prometheus_test.go
     835 ./pkg/client/monitoring/v1/zz_generated.deepcopy.go
    1152 ./pkg/prometheus/operator.go
   11410 ./pkg/client/monitoring/v1/openapi_generated.go
   24526 total

  -------------------------------------------------------------------------------
  Language                     files          blank        comment           code
  -------------------------------------------------------------------------------
  JSON                             9              0              0           6276
  Go                              31           1020           1065           6028

> I dunno, it just feels a little disingenuous, to me, to say that something that involves this much code is just a "packaging method" for a normal application.

What is disingenuous is to call the prometheus operator, that deploys an entire monitoring stack a "normal application" The operator sets up monitoring on all the nodes, runs all the server components including configuring grafana and setting up dashboards.

Meanwhile,

  $ cloc puppet-prometheus-5.0.0 puppet-grafana-4.2.0
     192 text files.
     157 unique files.
      48 files ignored.
  
  github.com/AlDanial/cloc v 1.76  T=1.09 s (131.9 files/s, 10534.0 lines/s)
  --------------------------------------------------------------------------------
  Language                      files          blank        comment           code
  --------------------------------------------------------------------------------
  Ruby                             43            425            108           2477
  Puppet                           32            132           1747           2280
  Markdown                          6            494              0           1201
  ERB                              20            105              0            925
  YAML                             35              0            200            459
  JSON                              4              0              0            329
  Bourne Shell                      2             28            130            198
  Bourne Again Shell                2             28             56            176
  --------------------------------------------------------------------------------
  SUM:                            144           1212           2241           8045
  --------------------------------------------------------------------------------

> I believe this is the kind of thing people actually want. Highly efficient, thin "VMs" that are easy to manage and run as independent systems without requiring the resource commitment.

Or, people want to use k8s so they can run entire clusters of machines as a single consistent system and take advantage of things like rolling deployments and self healing applications.

True. I certainly admit that I don't know the code well enough to know exactly which lines or files are critical and which aren't, but I think it's getting into the weeds to nitpick the specifics too much (for example, I would argue that while it's probably good to exclude generated files, you shouldn't exclude the tests from the line count since they require real human time to maintain).

The point is that thousands of lines of unique, application-specific code are required to create an "operator" that runs Prometheus within k8s. This is not what most people think of when someone says "a method for packaging".

> What is disingenuous is to call the prometheus operator, that deploys an entire monitoring stack a "normal application"

While Prometheus is certainly a big system in its own right, I don't think that necessarily makes it a bad representative. Many people are planning to port their own complicated systems to Kubernetes.

> Or, people want to use k8s so they can run entire clusters of machines as a single consistent system and take advantage of things like rolling deployments and self healing applications.

It's hard to talk about this because there is so much wrapped up into the gob that is k8s, and of course not all of it is bad. But we've had "rolling deployments" and "self-healing applications" before, without having to write 10k+ lines of code to manage the platform deployment. These aren't a new thing to k8s.

k8s provides a platform that gives a nomenclature to them, but it's not always clear that there is a benefit to running on that platform v. running more traditional setups, especially when you consider that you still have to configure and code your (k8s-internal) load balancers, web servers, and applications to handle these things.

There's no free lunch. Kubernetes is a container orchestrator. It automates system-level commands like "docker run ..." and provides a (mostly redundant) fabric for those containers to feed into. That's great and there are some people who really need that, but far too many people read comments like yours and interpret it to mean "If I use Kubernetes I will have self-healing applications". It doesn't work that way.

You continue to be disingenuous and imply that every application requires 10k lines of code to run on k8s.

I recently used k8s to deploy an application. To configure 2 services with exterior and interior load balancing and health checks for rolling deployments and self healing took 115 lines of yaml, maybe 40 of which was specific to my application.

115 lines. Not 10,000+

Then once things were working I created a 2nd namespace for production and deployed an entire 2nd copy of everything. This took me 10 minutes and 2 kubectl commands.

> "If I use Kubernetes I will have self-healing applications". It doesn't work that way.

But that's exactly how it worked. I wrote 115 lines of yaml and had multiple environments, load balancers, health checks, and rolling deployments.

I know how to do this using "traditional setups", and I know it takes a lot more than 115 lines of generic yaml.

Let me clarify and state unambiguously that it won't necessarily take 10k lines of code to run any random application on Kubernetes.

You can, in fact, deploy Prometheus without using the Prometheus operator and you'll technically be "running your monitoring" within k8s. It just isn't likely to be very reliable or useful. :)

> But that's exactly how it worked. I wrote 115 lines of yaml and had multiple environments, load balancers, health checks, and rolling deployments.

If you already had a fully "stateless", self-healing capable application running on not-k8s, and your layout is as simplistic as "2 services with load balancers", you can probably move to Kubernetes with a comparatively small amount of fuss. If your existing setup was pretty tiny, this may have been a worthwhile project.

If you didn't already have a stateless, self-healing-capable system, and you didn't change your application to accommodate it as part of the port, then regardless of what Kubernetes reports about your pod state, you don't have a self-healing application.

The barrier between application and platform is artificial. They must work together. It's sort of a convenient fantasy that you can try to demarcate these areas. You can't just take any random thing and throw it on Kubernetes and say it's all good now because you can watch k8s cycle your pods.

Maybe you think this is implicit, but as someone who has spent the last 2.5 years building out k8s clusters for software written by average developers, I can assure you that there are a great deal of people who aren't getting this message.

I went full-time freelance about a month ago. One of the last in-house k8s services I deployed, the guy told me, "Oh yeah, we can't run more than one instance of this, or it will delete everything." Yet, these people are very proud of the "crazy scalability" they get from running on Kubernetes. Hope the next guy reads the comments and doesn't nudge that replicas field!

If you already had a non-trivial system that worked well for failover, recovery, self-healing, etc., why'd you replace it with something that is, for example, still just barely learning how to communicate reliably with non-network-attached-storage, as a beta feature in 1.10 [0], released last month? There are many things that sysadmins take for granted that don't really work well within k8s.

I accept that at first glance and with superficial projects, it can be easy to throw the thing over the fence and let k8s's defaults deal with everything. This is definitely the model and the demographic that Google has been pursuing. But if you have something more serious going on, you still have to dig into the internals of nginx and haproxy within your k8s cluster. You still have to deal with DNS. You have to deal with all the normal stuff that is used in network operations, but now, you're just dealing with a weirdly-shaped vaguely-YAMLish version of it, within the Great Googly Hall of Mirrors.

Once you do that enough, you say "Well, why am I not just doing this through real DNS, real haproxy, real nginx, like we used to do? Why am I adding this extra layer of complication to everything, including the application code that has to be adapted for Kubernetes-specific restrictions, and for which I must write lines of code as an operator to ensure proper lifecycle behavior?"

Most people aren't willing to give themselves an honest answer to that question, partially because they don't really ask it. They just write some YAML and throw their code over the fence, now naively assured that the system is "self-healing". Then they get on HN and blast anyone who dares to question that experience.

[0] https://github.com/kubernetes/features/issues/121