Guide to OOMKill Alerting in Kubernetes Clusters

Another hidden issue is that as a container gets close to running out of memory, it furiously drops read only pages from memory, only to need to read some of them back into memory moments later.

This pathological swapping behavior can impact other workloads on the system.

cgroups2 has better protections against this behavior.

Re: Guide to OOMKill Alerting in Kubernetes Clusters

#6

Post by draganm » Mon, Nov 23, 2020, 11:21 PM UTC

geoffbp wrote: post #2

This fails to load for me

Sorry for that - seems I've under-estimated possible traffic. Scaled up the server a bit now.

Re: Guide to OOMKill Alerting in Kubernetes Clusters

#7

Post by throwii » Mon, Nov 23, 2020, 11:48 PM UTC

ec109685 wrote: post #5

Another hidden issue is that as a container gets close to running out of memory, it furiously drops read only pages from memory, only to need to read some of them back into memory moments later. This pathological swapping behavior can impact other workloads on the system. cgroups2 has better protections against this behavior.

Is there an issue on cgroups2 adoption for Kubernetes somewhere?

Re: Guide to OOMKill Alerting in Kubernetes Clusters

#8

Post by The_rationalist » Mon, Nov 23, 2020, 11:50 PM UTC

throwii wrote: post #7

ec109685 wrote: post #5

Another hidden issue is that as a container gets close to running out of memory, it furiously drops read only pages from memory, only to need to read some of them back into memory moments later. This pathological swapping behavior can impact other workloads on the system. cgroups2 has better protections against this behavior.

Is there an issue on cgroups2 adoption for Kubernetes somewhere?

https://github.com/kubernetes/enhancements/blob/master/keps/...

Re: Guide to OOMKill Alerting in Kubernetes Clusters

#9

Post by dilatedmind » Mon, Nov 23, 2020, 11:52 PM UTC

would it have been sufficient to alert on high memory usage? It might be reasonable to set an alert on say 70% rss. As long as the pod does not pass this threshold and die before a metric can be sampled.

that "no such file or directory" looks to be coming from building a dynamic executable on debian and trying to run it on alpine.

Re: Guide to OOMKill Alerting in Kubernetes Clusters

#10

Post by throwii » Mon, Nov 23, 2020, 11:53 PM UTC

The_rationalist wrote: post #8

throwii wrote: post #7

Earlier quoted context omitted.

Is there an issue on cgroups2 adoption for Kubernetes somewhere?

https://github.com/kubernetes/enhancements/blob/master/keps/...

Thank you!