Go, Containers, and the Linux Scheduler

This exists: https://github.com/lxc/lxcfs

lxcfs is a FUSE filesystem that mocks /proc by inferring cgroup values in a way that makes other applications and libraries work without having to care about whether it runs in a container (to the best of its ability - there are definitely caveats).

One such example is that /proc/uptime should reflect the uptime of the container, not the host; additionally /proc/cpuinfo reflects the number of CPUs as a combination of cpu.max and cpuset.cpus (whichever the lower bound is).

As others also mentioned, inferring the number of CPUs could also be done using the sched_getaffinity syscall - this doesn't depend on /proc/cpuinfo, so depending on the library you're using you might be in a pickle.

Delusion level: over 9900.

I'm yet to find a container that contains only the application and what it needs. Most of the time I find that they contain at least libc and libpthreads (which are already present on the host, so not needed). More often I find metric tonnes of garbage that was not necessary by any metric, but was just too hard to remove, so was allowed to stay.

When you set CPU reservations properly in your container definitions, they don't "crawl[] to a halt" when other containers demand more CPU than they did before. Every container is guaranteed to get the CPU it requested the moment it needs it, regardless of what other containers are using right up till that moment. So the key is to set the requested CPU to the minimum the application owner needs in the contended case. That's why I say that "requests are limits in reverse" - a request declared by container A effectively creates an implicit limit on container B when container A needs the CPU.

Perhaps you're concerned that not setting limits causes oversubscription on a node. But this is not so: when scheduling pods, the K8S scheduler only considers requests, not limits. If every container is forced to declare a CPU request - as it should - then K8S will not overprovision pods onto a node, and every container will be entitled to all the CPU it requested when needed.

Consider, too, that most applications are memory-bound and concurrency-bound, not CPU bound. And a typical web server is also demand-driven and load-balanced; it doesn't spontaneously consume CPU. For those kinds of applications, resource consumption is roughly consistent across all the containers, and the "pod running just find a moment ago...crawling to a halt" isn't a phenomenon you're going to see, as long as they haven't yet consumed all the CPU they requested. Limits hurt more than help especially for these kinds of workloads, because being able to burst gives them headroom to serve while a horizontal scale-out process (adding more pods) is taking place.

Most CPU-bound workloads tend to be batch workloads (e.g. ETL, map/reduce). Teams who run those on shared nodes usually just want to get whatever CPU they can. These jobs tend not to have strict deadlines and the owners don't usually panic when a worker pod has its CPU throttled on a shared node. And those who do care about getting all the CPU available are frequently putting these workloads on dedicated nodes, often even dedicated clusters. And the nodes tend to be ephemeral - launched just-in-time to run the job and then terminated when the job has completed.

As others have pointed out, putting CPU limits on containers is a contributing cause of nodes having lower CPU utilization than they could otherwise sustain. Every node costs money -- whether you bought it or rent it -- so underutilized nodes represent wasted capital or cloud spend. You want nodes to run as hot as possible to get the most out of your investment. If your CFO/cloud financial team isn't breathing down your neck when they see idle nodes, they ought to be.