Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Re: [PATCH] OOM_pardon, a.k.a. don't kill my xlock (2004)

lwn.net

Post reply on HN
Print view View original

41–50 of 103 posts

Re: Re: [PATCH] OOM_pardon, a.k.a. don't kill my xlock (2004)

#41

Post by sph »

SoftTalker wrote: post #22
LoganDark wrote: post #3

Earlier quoted context omitted.

This is only a viable answer when overcommit is disabled. The problem comes when overcommit is enabled and you find yourself in a position where many programs think they already have memory and yet there is none to give them. If you simply kill the first piece of code that encounters the end of available memory you might take down anything including the kernel itself. Nothing like statically allocating memory can wor…

OOM killer always felt like a band-aid on a severed artery to me. I've rarely seen a machine that got into OOM state really recover without a full reboot.

Why would a system break if you SIGKILL a process?

I’ve seen plenty of server log with OOM killing mariadb processes, and then being restarted automatically by systemd, often with no one noticing if not days later.

The thing that bogs down systems and often makes them unrecoverable is when a memory hungry process starts swapping. Good luck trying to SSH in. Swap is such a silly idea on servers - good to deal with pages no one accesses, catastrophic when you’re out of RAM and memory latencies suddenly become 4 or 5 orders of magnitude slower.

Re: Re: [PATCH] OOM_pardon, a.k.a. don't kill my xlock (2004)

#42

Post by hyperpape »

dooglius wrote: post #33
hyperpape wrote: post #26

I confess, this is very funny and the underlying situation is a bit absurd, but it's unclear what point Brouwer is making by pointing out the absurdity. There surely is something absurd about having to register specific processes as exempt from the OOM killer. But given that the OOM killer exists, and could kill xlock...how should that be fixed?

The point is that the OOM killer shouldn't exist and arguing about how to tweak it is addressing the wrong problem

But the second clause doesn't follow from the first!

I don't think Linux was plausibly going to remove the OOM killer in 2004 or later. So the right solution for Linux is very much to tweak it to be less painful.

Re: Re: [PATCH] OOM_pardon, a.k.a. don't kill my xlock (2004)

#43

Post by loeg »

rwmj wrote: post #28
bellowsgulch wrote: post #25

Earlier quoted context omitted.

I looked into this, and actually, it seems like maybe you can? https://man7.org/linux/man-pages/man5/proc_pid_oom_score_adj... So, in actuality, I think your assertion just taught us all something, because despite knowing that the OOM killer and that the Magic SysRq key[1] exists, I didn't know you could configure this as an input! [1]: https://en.wikipedia.org/wiki/Magic_SysRq_key

I'm aware of it, but it's awkward to use in practice. You have to track down all the FF processes, each time you run it, and adjust all their scores.

Maybe firefox could self-adjust, as a policy?

Re: Re: [PATCH] OOM_pardon, a.k.a. don't kill my xlock (2004)

#44

Post by nick__m »

rwmj wrote: post #28
bellowsgulch wrote: post #25

Earlier quoted context omitted.

I looked into this, and actually, it seems like maybe you can? https://man7.org/linux/man-pages/man5/proc_pid_oom_score_adj... So, in actuality, I think your assertion just taught us all something, because despite knowing that the OOM killer and that the Magic SysRq key[1] exists, I didn't know you could configure this as an input! [1]: https://en.wikipedia.org/wiki/Magic_SysRq_key

I'm aware of it, but it's awkward to use in practice. You have to track down all the FF processes, each time you run it, and adjust all their scores.

You could launch it as a systemd user target with OOMScoreAdjust=500 in the service section; weird and unconventional but wrapped in .desktop file it doesn't appear to be unwieldy.

Re: Re: [PATCH] OOM_pardon, a.k.a. don't kill my xlock (2004)

#45

Post by jitl »

rwmj wrote: post #28
bellowsgulch wrote: post #25

Earlier quoted context omitted.

I looked into this, and actually, it seems like maybe you can? https://man7.org/linux/man-pages/man5/proc_pid_oom_score_adj... So, in actuality, I think your assertion just taught us all something, because despite knowing that the OOM killer and that the Magic SysRq key[1] exists, I didn't know you could configure this as an input! [1]: https://en.wikipedia.org/wiki/Magic_SysRq_key

I'm aware of it, but it's awkward to use in practice. You have to track down all the FF processes, each time you run it, and adjust all their scores.

sounds like a job for a program

Re: Re: [PATCH] OOM_pardon, a.k.a. don't kill my xlock (2004)

#46

Post by bellowsgulch »

rwmj wrote: post #28
bellowsgulch wrote: post #25

Earlier quoted context omitted.

I looked into this, and actually, it seems like maybe you can? https://man7.org/linux/man-pages/man5/proc_pid_oom_score_adj... So, in actuality, I think your assertion just taught us all something, because despite knowing that the OOM killer and that the Magic SysRq key[1] exists, I didn't know you could configure this as an input! [1]: https://en.wikipedia.org/wiki/Magic_SysRq_key

I'm aware of it, but it's awkward to use in practice. You have to track down all the FF processes, each time you run it, and adjust all their scores.

Ah. Yes, that is awkward. Well, nonetheless, you taught me a new feature. Thanks!

Re: Re: [PATCH] OOM_pardon, a.k.a. don't kill my xlock (2004)

#47

Post by cwillu »

jml7c5 wrote: post #15
cwillu wrote: post #7

(2004)

Thanks. I was confused for a bit, given these days you can do echo "-1000" > /proc/ /oom_score_adj to disable OOM killing for a process. https://github.com/torvalds/linux/blob/master/include/uapi/l...

There's also /proc/sys/vm/panic_on_oom and /proc/sys/vm/oom_kill_allocating_task for other behaviours suggested in the comments.

Re: Re: [PATCH] OOM_pardon, a.k.a. don't kill my xlock (2004)

#48

Post by nemothekid »

While I have had my time fighting the OOM killer, I believe overcommit would have always won. To torture the metaphor a bit more, airlines have OOF mechanism - they just eject the overcommitted passengers before the plane takes off.

A passenger buying a ticket is malloc(), but passengers don't always utilize the seat (use the memory). Normally this works out fine, but occasionally, there are too many passengers. Thankfully though instead of executing a couple passengers they give you a voucher.

Re: Re: [PATCH] OOM_pardon, a.k.a. don't kill my xlock (2004)

#49

Post by kelnos »

hyperpape wrote: post #26

I confess, this is very funny and the underlying situation is a bit absurd, but it's unclear what point Brouwer is making by pointing out the absurdity. There surely is something absurd about having to register specific processes as exempt from the OOM killer. But given that the OOM killer exists, and could kill xlock...how should that be fixed?

I think part of it is that the design of screen lockers on X11 is just broken. If the locker crashes (or is killed), then the screen unlocks. Security-wise, it fails open. On Windows and macOS (and Wayland, using the ext-screen-lock protocol, coupled with sane compositor policy), that can't happen.

The right way for this to work is for the X server to have an extension that lets a screen locker say "hey, I'm locking the screen now", and the X server should respond to that by pretending that the screen locker client is the only client that exists: no other client gets input or gets to draw. And if the screen locker crashes (or is killed), the X server should just put itself into a permanently-locked state where it will never again send any input to anything, and won't ever draw anything except a blank screen. That's not a desirable situation, of course, but it's better than unlocking the screen.

Re: Re: [PATCH] OOM_pardon, a.k.a. don't kill my xlock (2004)

#50

Post by lokar »

I know this is not a popular / mainstream position, but I managed a very large fleet of systems this way:

- no system swap

- enough memory for core system services set aside in a cgroup for them to use

- by default, all prod service binaries load all code pages into ram at start, and lock them in (no paging out code pages at runtime)

- if needed (rare) services can mount some swap in their own cgroup, but very much discouraged

You need to know how much ram you are going to use, and actually stick to that. Very little is wasted in practice, and you don't have to deal with OOMs all the time. Everything is much more predictable.

Post reply on HN