Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Guide to OOMKill Alerting in Kubernetes Clusters

netice9.com

Post reply on HN
Print view View original

11–20 of 33 posts

Re: Guide to OOMKill Alerting in Kubernetes Clusters

#11

Post by draganm »

dilatedmind wrote: post #9

would it have been sufficient to alert on high memory usage? It might be reasonable to set an alert on say 70% rss. As long as the pod does not pass this threshold and die before a metric can be sampled. that "no such file or directory" looks to be coming from building a dynamic executable on debian and trying to run it on alpine.

as for the first question - that wouldn't be enough. AFAIK mmap-ed pages are part of RSS and it's quite usual for them to use up everything up to the memory limit (databases kind of rely on this 'feature'). None of that would provoke an OOMKill.

for the second comment - I've used images the author has published on Docker hub. Maybe there would've been a way to make it work, but if you take a look at the amount of code in missing-container-metrics, you will realise that I've used less time to write that than I would've spent debugging someone else's Docker build and golang code that is not really maintained.

Re: Guide to OOMKill Alerting in Kubernetes Clusters

#13

Post by linsomniac »

dilatedmind wrote: post #9

would it have been sufficient to alert on high memory usage? It might be reasonable to set an alert on say 70% rss. As long as the pod does not pass this threshold and die before a metric can be sampled. that "no such file or directory" looks to be coming from building a dynamic executable on debian and trying to run it on alpine.

I mean that's fine if you're ok with 30% wasted memory... We just recently had to tune some JVM and monitoring settings because we do the initial and max heap allocation to around 90% memory. There's very little else going on.

Re: Guide to OOMKill Alerting in Kubernetes Clusters

#16

Post by segmondy »

ec109685 wrote: post #5

Another hidden issue is that as a container gets close to running out of memory, it furiously drops read only pages from memory, only to need to read some of them back into memory moments later. This pathological swapping behavior can impact other workloads on the system. cgroups2 has better protections against this behavior.

I think the issue is that your nodes have swaps. Why will you have swap on container nodes? IMO, the idea with container management is to get predictability with resources. If you have 8gb on a node, you know that the containers get 8gb. You might not be able to tell exactly how based on how it's configured, but you know once they collectively use 8gb, that's it. Swap is going to mess up things really bad in ways you can't even predict.

Re: Guide to OOMKill Alerting in Kubernetes Clusters

#18

Post by johncolanduoni »

segmondy wrote: post #16
ec109685 wrote: post #5

Another hidden issue is that as a container gets close to running out of memory, it furiously drops read only pages from memory, only to need to read some of them back into memory moments later. This pathological swapping behavior can impact other workloads on the system. cgroups2 has better protections against this behavior.

I think the issue is that your nodes have swaps. Why will you have swap on container nodes? IMO, the idea with container management is to get predictability with resources. If you have 8gb on a node, you know that the containers get 8gb. You might not be able to tell exactly how based on how it's configured, but you know once they collectively use 8gb, that's it. Swap is going to mess up things really bad in ways you…

Even without swap enabled or any explicit memory mapping, read only pages from executables (code, read-only data) are mapped into the process’ address space and may be evicted. Unless you explicitly lock those into RAM they still behave somewhat like swapped memory does, except the pages don’t need to be written back.

Re: Guide to OOMKill Alerting in Kubernetes Clusters

#19

Post by chokeartist »

hendry wrote: post #17

Use serverless guys. k8s is a train wreck of needless complexity for 99% of developers.

While I acknowledge you probably have solved for your use-case... I can't help but hardcore LOL at your somewhat terse perspective! Dude... K8S just got mature!
Post reply on HN