31–40 of 111 posts
Couldn't the unresponsive kubectl issue be resolved by isolating cpus and controlling yourself where processes go?
If they saw the issue, then either they have not configured their nodes right, missconfigured them or perhaps run something very old?
I'm quite curious to see a proper test bench
I don't really understand why Buffer (or anyone else, for that matter) would choose to remove CPU limits from services where they are extremely important rather than upgrading to a kernel version that doesn't have this bug.
Is upgrading a kernel of a docker host that straight forward? I would worry to keep everything compatible, with a lot of testing before any upgrade of this kind. It looks like they're running k8s with kops, and the fix was merged just a few weeks ago.
Is this really right? "The danger of not setting a CPU limit is that containers running in the node could exhaust all CPU available." My assumptions have been: 1. cpu request tells you how much cpu a pod gets MINIMUM always, independently of how much other pods use it or not 2. on GKE you can't request 100% cpu due to google reserving cpu for the node 3. if you have hard limits, your cluster utilisation will be bad -…
Buffer's solution of having different flavors of node, onto which mutually compatible workloads are scheduled in isolation from incompatible ones, is a very reasonable thing to do, even if this particular case is a bit of a head-scratcher.
I don't really understand why Buffer (or anyone else, for that matter) would choose to remove CPU limits from services where they are extremely important rather than upgrading to a kernel version that doesn't have this bug.
Is upgrading a kernel of a docker host that straight forward? I would worry to keep everything compatible, with a lot of testing before any upgrade of this kind. It looks like they're running k8s with kops, and the fix was merged just a few weeks ago.
TL:DR; Remove limits if you’re running below 4.19 due to a bug present. Update AMI if running on EKS. I was expecting a discussion about CPU limits and all that is here is a workaround for a bug.
This is a more detailed post on the same thing - part two indicates changes have been back-ported to a number of kernel versions:
Linux-stable: 4.14.154+, 4.19.84+, 5.3.9+
Ubuntu: 4.15.0-67+, 5.3.0-24+
Redhat Enterprise Linux:
RHEL 7: 3.10.0-1062.8.1.el7+
RHEL 8: 4.18.0-147.2.1.el8_1+
CoreOS: v4.19.84+This seems like a bad trade-off, at least for 99% of us who haven’t been using Kubernetes in production for the last 5 years and manage it ourselves. Putting all the “user facing” services in a state where one of them consuming all the CPU could affect all the others feels like a disaster waiting to happen.
The number of times I’ve seen CPU limits kill off pods during even mild spikes and causing pretty much downtime and “disaster” is just as surprising. Work on autoscaling nodes instead, don’t use cpu limits.
We then got stuck in discussions around partial core allocation. We didn’t have that many jobs configured to use less than a full core, but it did impact our container packing.
I don't really understand why Buffer (or anyone else, for that matter) would choose to remove CPU limits from services where they are extremely important rather than upgrading to a kernel version that doesn't have this bug.
Is upgrading a kernel of a docker host that straight forward? I would worry to keep everything compatible, with a lot of testing before any upgrade of this kind. It looks like they're running k8s with kops, and the fix was merged just a few weeks ago.