91–100 of 111 posts
Does anyone know if this bug is present on Google Kubernetes Engine (GKE)?
Removing CPU limits seems like a bad idea now that there's a kernel fix. But putting that aside... I don't understand why pods without CPU limits would cause unresponsive kubelets. For a long time now Kubernetes has allocated a slice for system services. While pods without CPU limits are allowed to burst, they are still limited to the amount of CPU allocated to kubernetes pods. Run "systemd-cgls" on a node and you'll…
Actually, why? Sure those guys may starve the ones without limits but they won't starve each other just because Linux will simply time-share the processes. And for services on the critical path (what they turned it off for) that seems like correct behaviour.
DISCLAIMER: I work for Red Hat Consulting as an OpenShift/k8s consultant. This is such a bad idea. And I get that they're point is to reduce latency. But the point of k8s is describe your workload accurately and allow it to make decisions on your behalf. The no-brainer way fix this is to set the CPU Requests and Limits to the same value and add an HPA. Setting CPU Requests and Limits to the same value usually gives p…
What does "scale atomically" even mean? How does removing limits relate to horizontal vs vertical? HPA is based on request utilization, not limits, afaik.
What's your take on the arguments against limits in the comment at https://news.ycombinator.com/item?id=24356073 ?
DISCLAIMER: I work for Red Hat Consulting as an OpenShift/k8s consultant. This is such a bad idea. And I get that they're point is to reduce latency. But the point of k8s is describe your workload accurately and allow it to make decisions on your behalf. The no-brainer way fix this is to set the CPU Requests and Limits to the same value and add an HPA. Setting CPU Requests and Limits to the same value usually gives p…
How are the limits incorporated into scheduling? I assumed that was based on requests. What does "scale atomically" even mean? How does removing limits relate to horizontal vs vertical? HPA is based on request utilization, not limits, afaik. What's your take on the arguments against limits in the comment at https://news.ycombinator.com/item?id=24356073 ?
Vertical -> give more resources to the program
Horizontal -> run more instances of the program
Removing limits gives your pods more resources (scaling them vertically) whereas creating more pods creates more copies (scaling horizontally).
Assuming parent meant scaling by whole units with "scale atomically", that is you have one or two running programs, not "1.5" if you just give it 50% more resources.
DISCLAIMER: I work for Red Hat Consulting as an OpenShift/k8s consultant. This is such a bad idea. And I get that they're point is to reduce latency. But the point of k8s is describe your workload accurately and allow it to make decisions on your behalf. The no-brainer way fix this is to set the CPU Requests and Limits to the same value and add an HPA. Setting CPU Requests and Limits to the same value usually gives p…
How are the limits incorporated into scheduling? I assumed that was based on requests. What does "scale atomically" even mean? How does removing limits relate to horizontal vs vertical? HPA is based on request utilization, not limits, afaik. What's your take on the arguments against limits in the comment at https://news.ycombinator.com/item?id=24356073 ?
In the low latency trading world, these concerns are addressed by partitioning resources (for CPU, with affinities). This seems like a simpler mechanism that doesn’t require the kernel/daemon to track resource usage and to impose limits. I see only upsides to performance (bandwidth and latency) and availability by partitioning resources — so what are the benefits of the alternative, using limits, beyond being able to…
DISCLAIMER: I work for Red Hat Consulting as an OpenShift/k8s consultant. This is such a bad idea. And I get that they're point is to reduce latency. But the point of k8s is describe your workload accurately and allow it to make decisions on your behalf. The no-brainer way fix this is to set the CPU Requests and Limits to the same value and add an HPA. Setting CPU Requests and Limits to the same value usually gives p…
DISCLAIMER: I work for Red Hat Consulting as an OpenShift/k8s consultant. This is such a bad idea. And I get that they're point is to reduce latency. But the point of k8s is describe your workload accurately and allow it to make decisions on your behalf. The no-brainer way fix this is to set the CPU Requests and Limits to the same value and add an HPA. Setting CPU Requests and Limits to the same value usually gives p…
- https://github.com/kubernetes/kubernetes/issues/51135
- https://github.com/libero/reviewer/issues/1023
- https://medium.com/@betz.mark/understanding-resource-limits-...
- https://medium.com/omio-engineering/cpu-limits-and-aggressiv...
This doesn't seem as dangerous as is being suggested -- and in a world with the kernel bug and some separation of workloads it seems very viable.
Obviously, in a world without the kernel bug it makes much less sense to not set limits, but as far as scheduling goes, well-set requests (+/- a VPA[0]), with a HPA[1] should be enough to handle sudden increases in scale, and for truly large increases that are a complete surprise (otherwise you could have planned for it) elastic infrastructure via a cluster autoscaler[2].
[0]: https://github.com/kubernetes/autoscaler/tree/master/vertica...
[1]: https://kubernetes.io/docs/tasks/run-application/horizontal-...
[2]: https://github.com/kubernetes/autoscaler/tree/master/cluster...
Removing CPU limits seems like a bad idea now that there's a kernel fix. But putting that aside... I don't understand why pods without CPU limits would cause unresponsive kubelets. For a long time now Kubernetes has allocated a slice for system services. While pods without CPU limits are allowed to burst, they are still limited to the amount of CPU allocated to kubernetes pods. Run "systemd-cgls" on a node and you'll…
> Removing CPU limits seems like a bad idea now that there's a kernel fix. Actually, why? Sure those guys may starve the ones without limits but they won't starve each other just because Linux will simply time-share the processes. And for services on the critical path (what they turned it off for) that seems like correct behaviour.
Live and learn!
Earlier quoted context omitted.
Maybe I'm misunderstanding you, but I'm pretty sure CPU limits will only limit the amount of CPU used even if there is more available. It will not kill off the pod. Memory limits however will kill the pod if the pod uses more than the limit.
It is possible to get into this state. CPU starvation can be so severe that containers start failing their liveness probes and are killed. This is obviously very different than things like memory limits where the kernel OOMKills you, but will look similar to the untrained observer. Their app is serving 503s and the containers are in a restart loop -- looks like a Kubernetes problem. In general, the problem is that pe…