41–50 of 149 posts
I've created a static code analyzer for Kubernetes objects, called kube-score, that can identify and prevent many of these issues. It checks for resource limits, probes, podAntiAffinities and much more.
It's important to understand that CPU resource requests are used for scheduling and not for limiting. As the author suggests, this can be an issue when there is CPU contention, but on the other hand, it might not be. That's because memory limits are even more important than CPU requests when scheduling: most applications use far more memory as a proportion of overall host resources than CPU.
Let's take an example. Suppose we have a 64GB worker node with 8 CPUs in it. Now suppose we have a number of pods to schedule on it, each with a memory limit of 2GB and a CPU request of 1 millicore (0.001CPU). On this node, we will be able to accommodate 32 such pods.
Now suppose one of the pods gets busy. This pod can have all the idle CPU it wants! That's because it's a request and not a limit.
Now suppose all of the pods become fully CPU contended. The way the Linux scheduler works is that it will use the CPU request as a relative weight with respect to the other processes in the parent cgroup. It doesn't matter that they're small as an absolute value; what matters is their relative proportion. So if they're all 1 millicore, they will all get equal time. In this example, we have 32 pods and 8 CPUs, so under full contention, each will get 0.25 CPU shares.
So when I talk to customers about resource planning, I actually usually recommend that they start with low CPU reservation, and optimize for memory consumption until their workloads dictate otherwise. It does happen that particularly greedy pods are out there, but that's not the typical case - and for those that are, they will often allocate all of a worker's CPUs in which case you might as well dedicate nodes to them and forget about how to micromanage the situation.
Earlier quoted context omitted.
And they are probably not going to change their mind by reading an anonymous sarcastic comment on HN.
For someone with a short time scale, only trawling this thread, of course not. For someone young in this space, this comment and one hundred others -for and against- sift into hiso'er consciousness as part of the perceived zeitgeist of kubernetes within the larger community. Perhaps after several years, this person may have an intuition to avoid kubernetes in favor of separate docker or lxd containers. That associati…
In my opinion, the most common mistake is not in the article : using kubernetes when you don't need to. Kubernetes has a lot of pros or the papers but in practice it's not worth it for most small and medium companies.
Earlier quoted context omitted.
What would you suggest as an alternative, simpler form for docker deploy, running and managing? Docker-compose?
What do you mean by docker hosting? Kubernetes (and other related tools) are container orchestration/management tools. As if often the case in the management space, if you're just running at small scale, you may not need anything beyond container command line tools and some scripts. You could also use Ansible to automate.
Earlier quoted context omitted.
What do you mean by docker hosting? Kubernetes (and other related tools) are container orchestration/management tools. As if often the case in the management space, if you're just running at small scale, you may not need anything beyond container command line tools and some scripts. You could also use Ansible to automate.
Thanks we are running a few node servers, we now deploy command line. Dev we use docker-compose. But we are looking for a way to easily share our servers. We developed it for Amsterdam open source. Around 20-30 cities are in line to start using it. Doesn't have to be scalable, or have high availibility. Ease of deployment, easy way to update and basic security. All sysadmins are pushing for kubernetes, although for t…
You could look at the big cloud providers directly. OpenShift [I work at Red Hat] also has a few different types of managed offerings.
This is what I'm trying to convince my current company about. They want everything in a single cluster (prod, test, stage, qa).
Of course self hosting makes this more difficult to justify, since it is additional expenses for more machines.
What would a good liveness and readyness probe do for a rails app? What kind of work and metrics would these 2 endpoints do in my app?
For liveness probe I guess you could check if your service is accepting TCP connections? I don’t think there should ever be a reason for your service to outright refuse connections unless the main service process has crashed (in which case it’s best to let Kubernetes restart the container instead of having a recovery mechanism inside the container itself like supervisord or daemon tools).
"more tenants or envs in shared cluster" This is what I'm trying to convince my current company about. They want everything in a single cluster (prod, test, stage, qa). Of course self hosting makes this more difficult to justify, since it is additional expenses for more machines.
What would a good liveness and readyness probe do for a rails app? What kind of work and metrics would these 2 endpoints do in my app?
From the article: > The other one is to tell if during a pod's life the pod becomes too hot handling too much traffic (or an expensive computation) so that we don't send her more work to do and let her cool down, then the readiness probe succeeds and we start sending in more traffic again.
Well... maybe. Is it a routine occurrence that an individual Pod becomes "too hot"? If your load balancer can retry a request on, say, a 503 Service Unavailable, you may be better off relying on that retry combined with CPU-based autoscaling to add another Pod (it's simpler, tradeoff is the load balancer may spend too much time retrying).
If you can't or don't want to add additional Pods, then your client is going to see that 503 (or similar) anyway. I'd say, then, that the point of a Pod claiming it's "not ready" to get itself removed from the load balanced pool is to allow the load balancer to more quickly find an available Pod, but this adds complexity and may be irrelevant if you run enough Pods to have some overhead capacity.
A Rails app is a bit different from a node/go/java app in that (typically at least, if you're using Unicorn or other forking servers) each individual Pod can only handle a limited number of concurrent requests (8, 16, whatever it is). It's more likely then that any given Pod is at capacity.
But, liveness/readiness are not so simple. If these probes go through the main application stack, then they're tying up one of the precious few worker processes, even if only momentarily. I haven't worked with Ruby in a number of years, but I remember running a webrick server in the unicorn master process, separate from the main app stack, to respond to these checks. But I did not implement a readiness check that tracked the number of requests and reported "not ready" if all the workers are busy.