So I actually answered this question for you when you asked it in December:
https://news.ycombinator.com/item?id=13241681 . ;)
Not to toot the same horn.
I have more experience with Kubernetes now than I did then, though I think a lot of those interface snafus, which is most of what the old post discussed, are still valid complaints.
You mentioned back then that if people aren't aware of some kube featureset, they end up reimplementing it. You mentioned logging. Can you tell me what you recommend for logging?
Right now, we have everything writing logs to stdout in the container, which gets recorded by kube in something like /var/log/kube/containers/ * . Then we have a fluentd container that reads the logs and uploads them to an external machine, which is running fluentd containers that receive the input stream, transform it according to rules, and pipe it out to a cloud-based log aggregator.
Is that how it's supposed to work? I know we used to do it somewhat differently, but had trouble with resource consumption that was disrupting our pods.
And here's a potentially legitimate Kubernetes issue: the scheduler is only reading the load generated from containers administered by Kubernetes, so even though the box is being thrashed, k8s will not be aware of the problem and continues to report the node's resources as healthy. In our case, the dockerd process was using 4.5 cores. Kubernetes seemed unaware and it was affecting our performance because another busy pod was on the server and was not getting clamped by its quota (since it, technically, was not hitting the quota for itself). Shouldn't these be defined relative to overall system resources, and not just the containers that kube can see?
It's possible Kubernetes was aware, but that we didn't know how to see it. A colleague told me the interface I was looking at for pod performance data reflects only the quotas, not the actual utilization, so maybe k8s knew (why isn't there a `kubetop` that can be run on the host to supervise?). I also hear that automatic rescheduling is still in draft stage, so if k8s detects high load on a kubelet, instead of moving the pod, it will just not assign new pods to it.
About addressing. Right now, we have dev, stage, and prod. We have a container that takes all traffic targeting the kubelet on port 80 and proxies it out to the pod using servicelb. It's set up to forward from http://app-label.env-name.example.com to a pod matching that label. It's annoying that we have to run an external container for this in the first place and do extra configuration on it, but it's also annoying that since servicelb is not aware of the beta label attempting to simulate hostnames, we can't individually address instances of apps. Is that how this is supposed to work, or is there a ready-built solution that makes this much easier that we're overlooking?
----
>I saw some of your other comments - I'd love to understand better what we could do to either a) make it feel like not an internal tool and b) if you didn't use Kubernetes or containers, how you'd prefer to run distributed workloads.
The best way to improve Kubernetes would be to:
a) simplify the terminology;
b) simplify the kubectl command, and particularly make it follow semi-standard Unix utility conventions (examples: -f should mean force or not be used (and yes, I know some other tools commit this sin too, but the association with "force" is nerve-wracking on a production environment), should probably be -r or -o; --from-file is ambiguous, and should be --read-literal or -r --raw or something like that);
c) simplify the configuration syntax and files;
d) provide simple ways to get real storage and real IPs (I haven't thoroughly investigated StatefulSets, so it may do this). I know that Google has long ago gone into a plane where these are irrelevant, but normal companies haven't, and IMO there's no reason they necessarily should;
e) provide simple backup and restoration methods (afaik, this involves multiple steps, including backing up the etcd cluster (not allowed to run on the master, or just conventionally doesn't?)).
My suggestion for distributed workloads is:
a) script your machine in Ansible or similar, and make a base image from that;
b) use your cloud provider of choice to deploy new instances under the circumstances considered necessary;
c) register the new node with haproxy or whatever sits at your front end (often a cloud-provided LB);
d) use conventional administrative and monitoring tooling to administer.
All of that is fully scriptable, in far less time than it takes to convert to Docker/k8s.
-----
>TBC, I totally agree with you - if you're running 1-2 nodes, and don't care about downtime, you should NEVER use Kubernetes or any orchestrator. Once you get to 3+, however, I can't imagine using anything else.
Why is the above-recommended distributed systems strategy inadequate, even for people with more than 3 nodes who care about downtime? What's fundamentally wrong with it? What unique value does k8s bring to the table?
What's the k8s replacement for htop? tcpdump? df? and all the other utilities, and why are they better, and what benefit does k8s provide, that we didn't already have, that warrants giving up everything? I know it's not TRYING to replace all of that per se, but the practicalities make kubernetes the authority for such information, and you're not supposed to have to get onto the kubelet itself to diagnose these types of issues so you can't even use root on the Docker host to try and do some of this.
In our deployment, the main k8s jockey just kills pods if they're acting up and hopes that will magically fix it, because the devs don't want to get near the setup with a 10-foot pole. Before, they would SSH in and go over the issues. Now, since not only can't you SSH in but there's a convoluted process to get a shell, and then the container will not have any diagnosis tools in it anyway, it's very hard to have them collaboratively troubleshoot a problem we're seeing in the wild.
I can't tell you how many times I've had someone tell me "They must have a better answer for that; Google uses it after all...". These people are under the mistaken impression that if Google is using it, it must be robust and stable, when in fact the opposite is usually true, and I don't think Google tries to pretend it's not. It's probably just misinterpreted the signal from the developer community as meaningful approbation, when it's really just blindly following the cool kid on the block.
It's not that k8s isn't a neat thing. It's just that it's solving a problem that pretty much only Google had, and now everyone else is plummeting down the rabbit hole.
We've spent probably 1.5 man-years getting our infrastructure containerized and kubed up. We're not at 100% in prod yet but getting pretty close. It's clear that I'm really dumb but I haven't even been the main one doing this, so can't blame it on me. That's a huge project, and how does k8s/Docker justify the cost? It lets us cut down on AWS usage? Sure, but nowhere near the proportion of cost. It "makes it easier" to administer the cluster? Nope, not a chance. It's fun to kill pods but that's living on the edge, most apps written by non-Google can't take that. Solution? Write an "operator", which appears to be any random program that intercepts the k8s API and does stuff to support pods, etc. That sucks, why don't I just keep the init script that works fine?
One could argue that k8s is about resource utilization, which is sort-of-but-not-really-true. And that has a downside; we've essentially turned our environment into a shared host now, and despite k8s quotas, we still have pods that are bad neighbors; this is turning out to be quite difficult to control.
So I'm just really at a loss here. Google wanted it and they built it and it's working for them, that's great. That's an internal tool. How does this help anyone who wants to run a normal site this way? I haven't heard a single success story that had a moral or goal behind it other than "I'm cool like Google now too."
And on top of all of this, before k8s even comes into the picture, you have to make a Dockerfile and run your application in Docker, which is a PITA itself, in terms of stability, security, and configuration (see above-mentioned dockerd hammering the box, dockerd hangs or breakages that render a box useless, having to deal with port forwards, name collisions, large Docker caches, docker's inability to remove old images or containers on its own, Dockerfile lameness, culture of importing unknown images, etc.).
I like the ideas of containers (and have liked them when they were called "jails" in past eras ;) ) and I like the idea of orchestrators like kubernetes. But I think it's very early days right now for both, and that they will need radically improved simplification and stability before they are appropriate for general use.