Guide to OOMKill Alerting in Kubernetes Clusters

Post by dharmab » Tue, Nov 24, 2020, 4:54 PM UTC

m1keil wrote: post #26

jacques_chester wrote: post #22

Earlier quoted context omitted.

I don't think it would work to report an OOMkill. The process that would answer the healthcheck probe would be gone.

It's not clear to me what happens after the OOM. Does the init process restarts the daemon? I would argue that it shouldn't. If the process stops responding to a healthcheck, it's the scheduler's responsibility (k8s in this case) to handle it. Crashes in this case should be handled in a similar way, whether it's due to OOM or a bug. Maybe I'm missing another scenario here?

Post by ec109685 » Thu, Nov 26, 2020, 11:44 PM UTC

pflanze wrote: post #29

ec109685 wrote: post #5

Another hidden issue is that as a container gets close to running out of memory, it furiously drops read only pages from memory, only to need to read some of them back into memory moments later. This pathological swapping behavior can impact other workloads on the system. cgroups2 has better protections against this behavior.

How can you use cgroups2 for this? I know that BSD resource limits are basically useless for this as they only allow to limit virtual memory, not RSS use. EarlyOOM [1] is a configurable daemon that kills processes early enough to (hopefully) prevent thrashing. I'm using it on my Linux desktops (it has proven to catch my own programs' runaway memory usage before it risks locking up the development machine), but it may…