Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

There is an OOM kill count in Linux

medium.com

Post reply on HN
Print view View original

41–50 of 66 posts

Re: There is an OOM kill count in Linux

#41

Post by Dalewyn »

samus wrote: post #33
magicalhippo wrote: post #16

Earlier quoted context omitted.

This is one area where Windows is far superior to Linux as a desktop experience. By accident, I once wrote an infinite loop that just allocated a bunch of memory. As I ran the program my Windows system quickly became laggy and unresponsive. However, not completely. With a bit of patience I got Task Manager up and managed to kill the process and the system was back to normal within a minute. On Linux, I experience the…

Since Linux has Unix-legacy and is usually intended to be used on servers, there is one well-intended assumption at play here: unless different settings are applied, all processes are in principle created equal. This also applies to GUI applications, and means that a single out-of-control application can lock up the whole system. That's actually fine on multi-user systems. A competent admin would set up tight limits…

Explorer and Task Manager are both shells (along with their CLI sibling shells CMD and Powershell), but Task Manager is special in that Windows Logon is always actively monitoring for CTRL+ALT+DEL.

Windows Logon is superior to all the shells as a core Windows subsystem in that it is the one that executes the shell (usually Explorer) on boot/login and can also execute Task Manager, in addition to its primary duties of tracking and securing user logins (hence the name).

This means even if Explorer has completely froze or has crashed, you can always get to Task Manager via Windows Logon and then kill or execute another instance of Explorer (or CMD or Powershell!) or otherwise force reboot the system from Windows Logon.

Re: There is an OOM kill count in Linux

#42

Post by alpaca128 »

roblabla wrote: post #34
gnramires wrote: post #23

Earlier quoted context omitted.

Indeed! I wonder why the system can't just stop giving the process memory when it's starting to become critical? (say, you're over 95% memory used) I really think a more general solution is treating memory (and CPU time as welL!) as a scarce resource, and programs should either: (1) Deal with having severely denied resources (default behavior); (2) Use a communication protocol to negotiate memory with the system. Neg…

> Maybe this is reinventing the wheel? I believe something like that exists on mobile platforms for memory, at least on iOS you get a message (applicationDidReceiveMemoryWarning:) when the system is memory starved and wants your app to free memory. If you don't release enough memory and the memory pressure doesn't go down, the system will start killing apps.

And on Mac OS you'll get a popup dialog that lets you select programs you want to force quit to free memory. Which is a nice idea as everything else stays responsive, unfortunately it doesn't list any system processes so it's useless if one of them has a bug - which is the reason I even learned of the existence of this oom dialog.

Re: There is an OOM kill count in Linux

#43

Post by larve »

contrarian1234 wrote: post #7

Out of curiosity, what do people recommend for OOM these days on Linux? My number one recurring issue on Ubuntu is me not noticing the memory usage, running out of memory and locking up my system (forcing a hard reboot and loss of unsaved work) EDIT: I'd preferably like an OOM handler that would freeze my system and pop up a little menu from which I could select which process to nuke

In a previous embedded job, I built a daemon watching a couple of “at risk of eating too much memory” processes that were not critical for the actual function of the machine, and killed them proactively when memory started getting low. This prevented the system from going into starvation and spending 10 seconds doing whatever each time an Ethernet packet came in.

As for my pragmatic solution at home: max out the physical RAM…

Re: There is an OOM kill count in Linux

#44

Post by hnfong »

gnramires wrote: post #23
magicalhippo wrote: post #16

Earlier quoted context omitted.

This is one area where Windows is far superior to Linux as a desktop experience. By accident, I once wrote an infinite loop that just allocated a bunch of memory. As I ran the program my Windows system quickly became laggy and unresponsive. However, not completely. With a bit of patience I got Task Manager up and managed to kill the process and the system was back to normal within a minute. On Linux, I experience the…

Indeed! I wonder why the system can't just stop giving the process memory when it's starting to become critical? (say, you're over 95% memory used) I really think a more general solution is treating memory (and CPU time as welL!) as a scarce resource, and programs should either: (1) Deal with having severely denied resources (default behavior); (2) Use a communication protocol to negotiate memory with the system. Neg…

Most software doesn't really handle memory allocation failures too well.

malloc() can fail, everyone knows this in theory, but assume you're the programmer handling the malloc() error -- you know the system is likely to have already run out of free memory -- how do you make the program fail gracefully without doing anything at all to allocate memory?

It's not even safe to call printf() since it might have to allocate a new string. There are very few things in modern code that doesn't "incidentally" allocate a couple bytes here and there.

Not to mention languages higher level than C where you don't directly call malloc() but have the language handle it for you (GC etc.). Handling OOM for those programs is even more impossible because you really have no idea whatever class/method/function you're calling is going to trigger an allocation or not.

The best you can do in those cases is try to commit critical data (if any) to disk and have the program die ASAP. Which is not that different from having the kernel kill it for you.

Re: There is an OOM kill count in Linux

#45

Post by seba_dos1 »

marcodiego wrote: post #37
contrarian1234 wrote: post #7

Out of curiosity, what do people recommend for OOM these days on Linux? My number one recurring issue on Ubuntu is me not noticing the memory usage, running out of memory and locking up my system (forcing a hard reboot and loss of unsaved work) EDIT: I'd preferably like an OOM handler that would freeze my system and pop up a little menu from which I could select which process to nuke

Is this still the case on a modern distro using MGLRU and systemd-oomd?

Not really, but not all distros have caught up yet.

Re: There is an OOM kill count in Linux

#46

Post by weebull »

hnfong wrote: post #44
gnramires wrote: post #23

Earlier quoted context omitted.

Indeed! I wonder why the system can't just stop giving the process memory when it's starting to become critical? (say, you're over 95% memory used) I really think a more general solution is treating memory (and CPU time as welL!) as a scarce resource, and programs should either: (1) Deal with having severely denied resources (default behavior); (2) Use a communication protocol to negotiate memory with the system. Neg…

Most software doesn't really handle memory allocation failures too well. malloc() can fail, everyone knows this in theory, but assume you're the programmer handling the malloc() error -- you know the system is likely to have already run out of free memory -- how do you make the program fail gracefully without doing anything at all to allocate memory? It's not even safe to call printf() since it might have to allocate…

The "attempt to save critical data" is a pretty major difference.

Re: There is an OOM kill count in Linux

#47

Post by lokar »

ynik wrote: post #21
gkbrk wrote: post #10

Earlier quoted context omitted.

Swap usually makes it worse, without swap there is some chance that the Linux OOM killer does something useful and saves the system. With swap, it becomes a frozen system that never manages to kill anything due to all the swapping. You can wait 5 minutes, 10 minutes, or 15 minutes, but the system never recovers without a reboot.

That only works if the system is accessing mostly anonymous pages. If the load on the system is accessing plenty of mmapped code/data pages, it can still trash those even if swap is disabled. I've still seen systems hanging for 10+ minutes without recovering even though swap was already disabled. The Linux kernel OOM killer only acts if there's nothing left that can be discarded, which often happens way too late to s…

This is why on production systems important binaries should mlock all of their code pages into ram at startup.

Re: There is an OOM kill count in Linux

#48

Post by [deleted] »

mgdlbp wrote: post #28
contrarian1234 wrote: post #7

Out of curiosity, what do people recommend for OOM these days on Linux? My number one recurring issue on Ubuntu is me not noticing the memory usage, running out of memory and locking up my system (forcing a hard reboot and loss of unsaved work) EDIT: I'd preferably like an OOM handler that would freeze my system and pop up a little menu from which I could select which process to nuke

Enable Alt+SysRq+F and do it manually. Works in seconds every time; the kernel can take days because it's not eager enough when thrashing. Can have security implications, but the target is chosen in the typical way and screen locks should be setting a score_adj to avoid being picked.

[deleted]

Re: There is an OOM kill count in Linux

#49

Post by kaba0 »

Dalewyn wrote: post #41
samus wrote: post #33

Earlier quoted context omitted.

Since Linux has Unix-legacy and is usually intended to be used on servers, there is one well-intended assumption at play here: unless different settings are applied, all processes are in principle created equal. This also applies to GUI applications, and means that a single out-of-control application can lock up the whole system. That's actually fine on multi-user systems. A competent admin would set up tight limits…

Explorer and Task Manager are both shells (along with their CLI sibling shells CMD and Powershell), but Task Manager is special in that Windows Logon is always actively monitoring for CTRL+ALT+DEL. Windows Logon is superior to all the shells as a core Windows subsystem in that it is the one that executes the shell (usually Explorer) on boot/login and can also execute Task Manager, in addition to its primary duties of…

Linux has something similar to ctrl+alt+del monitoring in the form of REISUB and alia. But it is much more crude (and nowadays mostly disabled by default).

Re: There is an OOM kill count in Linux

#50

Post by kaba0 »

roblabla wrote: post #34
gnramires wrote: post #23

Earlier quoted context omitted.

Indeed! I wonder why the system can't just stop giving the process memory when it's starting to become critical? (say, you're over 95% memory used) I really think a more general solution is treating memory (and CPU time as welL!) as a scarce resource, and programs should either: (1) Deal with having severely denied resources (default behavior); (2) Use a communication protocol to negotiate memory with the system. Neg…

> Maybe this is reinventing the wheel? I believe something like that exists on mobile platforms for memory, at least on iOS you get a message (applicationDidReceiveMemoryWarning:) when the system is memory starved and wants your app to free memory. If you don't release enough memory and the memory pressure doesn't go down, the system will start killing apps.

Aren’t it simply more common for apps to go into a “save state and basically exit” mode, from which they are expected to recover without any fuss? Like, ios is very aggressive in “killing” background apps, and since this is the general consensus, apps just behave nicely the majority of time.
Post reply on HN