Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

There is an OOM kill count in Linux

medium.com

Post reply on HN
Print view View original

21–30 of 66 posts

Re: There is an OOM kill count in Linux

#21

Post by ynik »

gkbrk wrote: post #10
manymatter wrote: post #9

Earlier quoted context omitted.

I've never had that problem. Maybe you don't setup much swap space as I would expect the system to run well enough to save your work and shutdown for most common leaks.

Swap usually makes it worse, without swap there is some chance that the Linux OOM killer does something useful and saves the system. With swap, it becomes a frozen system that never manages to kill anything due to all the swapping. You can wait 5 minutes, 10 minutes, or 15 minutes, but the system never recovers without a reboot.

That only works if the system is accessing mostly anonymous pages. If the load on the system is accessing plenty of mmapped code/data pages, it can still trash those even if swap is disabled. I've still seen systems hanging for 10+ minutes without recovering even though swap was already disabled.

The Linux kernel OOM killer only acts if there's nothing left that can be discarded, which often happens way too late to save the system. You need a user-mode OOM-killer like earlyoom if you want to keep the system responsive.

Re: There is an OOM kill count in Linux

#22

Post by grumbel »

manymatter wrote: post #14
gkbrk wrote: post #10

Earlier quoted context omitted.

Swap usually makes it worse, without swap there is some chance that the Linux OOM killer does something useful and saves the system. With swap, it becomes a frozen system that never manages to kill anything due to all the swapping. You can wait 5 minutes, 10 minutes, or 15 minutes, but the system never recovers without a reboot.

That would normally mean you ignore the system getting slower for a long time.

You have only about 10sec between the system getting slower and the system locking up completely. If you manage to hit the Magic SysRq key combination to trigger OOM manually, that can save the system, but you have to be quick.

[1] https://en.wikipedia.org/wiki/Magic_SysRq_key

Re: There is an OOM kill count in Linux

#23

Post by gnramires »

magicalhippo wrote: post #16
contrarian1234 wrote: post #7

Out of curiosity, what do people recommend for OOM these days on Linux? My number one recurring issue on Ubuntu is me not noticing the memory usage, running out of memory and locking up my system (forcing a hard reboot and loss of unsaved work) EDIT: I'd preferably like an OOM handler that would freeze my system and pop up a little menu from which I could select which process to nuke

This is one area where Windows is far superior to Linux as a desktop experience. By accident, I once wrote an infinite loop that just allocated a bunch of memory. As I ran the program my Windows system quickly became laggy and unresponsive. However, not completely. With a bit of patience I got Task Manager up and managed to kill the process and the system was back to normal within a minute. On Linux, I experience the…

Indeed! I wonder why the system can't just stop giving the process memory when it's starting to become critical? (say, you're over 95% memory used)

I really think a more general solution is treating memory (and CPU time as welL!) as a scarce resource, and programs should either: (1) Deal with having severely denied resources (default behavior); (2) Use a communication protocol to negotiate memory with the system.

Negotiation could mean (a) The process freeing memory spontaneously when there's pressure (another process requires it) and it's inactive, (b) Requesting the system free memory from other processes when user requires.

This would impose more memory management burden on developers (trying to fulfill OS requests), but in turn it would make for a far better experience when there's memory contention.

Disclaimer: I have no idea how memory allocation works in detail. Maybe this is reinventing the wheel?

Re: There is an OOM kill count in Linux

#24

Post by gkbrk »

ynik wrote: post #21
gkbrk wrote: post #10

Earlier quoted context omitted.

Swap usually makes it worse, without swap there is some chance that the Linux OOM killer does something useful and saves the system. With swap, it becomes a frozen system that never manages to kill anything due to all the swapping. You can wait 5 minutes, 10 minutes, or 15 minutes, but the system never recovers without a reboot.

That only works if the system is accessing mostly anonymous pages. If the load on the system is accessing plenty of mmapped code/data pages, it can still trash those even if swap is disabled. I've still seen systems hanging for 10+ minutes without recovering even though swap was already disabled. The Linux kernel OOM killer only acts if there's nothing left that can be discarded, which often happens way too late to s…

Yes, I don't disagree that user-mode OOM-killers are helpful, or that the system can still hang without swap.

I'm just saying turning on swap, or increasing the swap capacity does not fix the problem, and it usually makes it worse.

Re: There is an OOM kill count in Linux

#26

Post by Toorkit »

contrarian1234 wrote: post #7

Out of curiosity, what do people recommend for OOM these days on Linux? My number one recurring issue on Ubuntu is me not noticing the memory usage, running out of memory and locking up my system (forcing a hard reboot and loss of unsaved work) EDIT: I'd preferably like an OOM handler that would freeze my system and pop up a little menu from which I could select which process to nuke

I recently installed Linux on my main desktop again. After freezing up twice in a week, I looked around.

I enabled systemd-oom, but it hasn't been necessary, because I also enabled zram as a swap device. I've seen it compress multiple gb of ram into a few hundred mb and haven't frozen my desktop since.

Re: There is an OOM kill count in Linux

#27

Post by seba_dos1 »

contrarian1234 wrote: post #7

Out of curiosity, what do people recommend for OOM these days on Linux? My number one recurring issue on Ubuntu is me not noticing the memory usage, running out of memory and locking up my system (forcing a hard reboot and loss of unsaved work) EDIT: I'd preferably like an OOM handler that would freeze my system and pop up a little menu from which I could select which process to nuke

You want to enable thrashing prevention: https://docs.kernel.org/next/admin-guide/mm/multigen_lru.htm...

Make sure you use zram (or zswap) as well. May also consider enabling userspace oom handler like others already suggested, but it's less important than the rest.

Re: There is an OOM kill count in Linux

#28

Post by mgdlbp »

contrarian1234 wrote: post #7

Out of curiosity, what do people recommend for OOM these days on Linux? My number one recurring issue on Ubuntu is me not noticing the memory usage, running out of memory and locking up my system (forcing a hard reboot and loss of unsaved work) EDIT: I'd preferably like an OOM handler that would freeze my system and pop up a little menu from which I could select which process to nuke

Enable Alt+SysRq+F and do it manually. Works in seconds every time; the kernel can take days because it's not eager enough when thrashing.

Can have security implications, but the target is chosen in the typical way and screen locks should be setting a score_adj to avoid being picked.

Re: There is an OOM kill count in Linux

#29

Post by wakeupcall »

contrarian1234 wrote: post #7

Out of curiosity, what do people recommend for OOM these days on Linux? My number one recurring issue on Ubuntu is me not noticing the memory usage, running out of memory and locking up my system (forcing a hard reboot and loss of unsaved work) EDIT: I'd preferably like an OOM handler that would freeze my system and pop up a little menu from which I could select which process to nuke

I'm using earlyoom on desktop, systemd-oomd on servers.

Due to cgroupv2, systemd-oomd tends to kill entire process groups instead of a single run-away process [1]. It's perfect for servers where almost every process is part of a system service already.

For desktop usage I found this behavior unacceptable. I would sometimes get my entire X session killed when trying something dumb (which is the reason I run such a daemon in the first place), or my editor when spawning a helper subprocess...

earlyoom was the simplest daemon of the bunch I tried, and worked out of the box without special configuration.

[1] Or at least this was the state ~1yr ago.

Re: There is an OOM kill count in Linux

#30

Post by grumbel »

contrarian1234 wrote: post #7

Out of curiosity, what do people recommend for OOM these days on Linux? My number one recurring issue on Ubuntu is me not noticing the memory usage, running out of memory and locking up my system (forcing a hard reboot and loss of unsaved work) EDIT: I'd preferably like an OOM handler that would freeze my system and pop up a little menu from which I could select which process to nuke

More RAM. It's the only thing that can reliably prevent it. Magic SysRq key[1] can trigger the OOM manually, which can sometimes rescue the system if you notice it soon enough. But if you run into the situation regularly under normal use, not programs running amok, more RAM is a must.

If you can't upgrade the RAM, zram[2] is also worth a try.

[1] https://en.wikipedia.org/wiki/Magic_SysRq_key

[2] https://en.wikipedia.org/wiki/Zram

Post reply on HN