Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Modernizing Linux swapping: introducing the swap table

lwn.net

Post reply on HN
Print view View original

101–110 of 132 posts

Re: Modernizing Linux swapping: introducing the swap table

#101

Post by man8alexd »

AtlasBarfed wrote: post #99
man8alexd wrote: post #33

Earlier quoted context omitted.

https://chrisdown.name/2018/01/02/in-defence-of-swap.html

I get we want a magical OS that reads minds and abstracts itself from users, but these days low mem situations are technical events for people trying to optimize resources use and balance uptime. So not being able to mark apps processes as kill me first and leave others like ssh bash up is a missing feature. Shouldn't the os have a bunch of auto hooks to invoke for machines under duress? Yes you might be able to do i…

> So not being able to mark apps processes as kill me first and leave others like ssh bash up is a missing feature.

systemd OOMScoreAdjust has existed for at least a decade. /proc/*/oom_adj since Linux kernel 2.6.11 - two decades. cgroups are also two decades old. SSH by default has OOMScoreAdjust=-1000 and is entirely protected from OOM.

Re: Modernizing Linux swapping: introducing the swap table

#102

Post by AtlasBarfed »

man8alexd wrote: post #55
dspillett wrote: post #46

Earlier quoted context omitted.

> There’s a common rule of thumb that says you should have swap space equal to some multiple of your RAM. That rule came about when RAM was measured in a couple of MB rather than GB, and hasn't made sense for a long time in most circumstances (if you are paging our a few GB of stuff on spinning drives your system is likely to be stalling so hard due to disk thrashing that you hit the power switch, and on SSDs you are…

Swap is not a replacement for RAM. It is not just slow. It is very-very-very slow. Even SSDs are 10^3 slower at random access with small 4K blocks. Swap is for allocated but unused memory. If the system tries to use swap as active memory, it is going to become unresponsive very quickly - 0.1% memory excess causes a 2x degradation, 1% - 10x degradation, 10% - 100x degradation.

What is allocated but unused memory? That sounds like memory that will be used in the near future and we are scheduling in an annoying disk load when it is needed

You are of course highlighting the problem that virtual addressing was intended to over abstract memory resource usage, but it provides poor facilities for power users to finely prioritize memory usage.

The example of this is game consoles, which didn't have this layer. Game writers had to reserve parts of ram fur specific uses.

You can't do this easily in Linux afaik, because it is forcing the model upon you.

Re: Modernizing Linux swapping: introducing the swap table

#103

Post by man8alexd »

AtlasBarfed wrote: post #102
man8alexd wrote: post #55

Earlier quoted context omitted.

Swap is not a replacement for RAM. It is not just slow. It is very-very-very slow. Even SSDs are 10^3 slower at random access with small 4K blocks. Swap is for allocated but unused memory. If the system tries to use swap as active memory, it is going to become unresponsive very quickly - 0.1% memory excess causes a 2x degradation, 1% - 10x degradation, 10% - 100x degradation.

What is allocated but unused memory? That sounds like memory that will be used in the near future and we are scheduling in an annoying disk load when it is needed You are of course highlighting the problem that virtual addressing was intended to over abstract memory resource usage, but it provides poor facilities for power users to finely prioritize memory usage. The example of this is game consoles, which didn't hav…

Unused or Inactive memory is memory that hasn't been accessed recently. The kernel maintains LRU (least recently used) lists for most of its memory pages. The kernel memory management works on the assumption that the least recently used pages are least likely to be accessed soon. Under memory pressure, when the kernel needs to free some memory pages, it swaps out pages at the tail of the inactive anonymous LRU.

Cgroup limits and OOM scores allow to prioritize memory usage on a per-process and per-process group basis. madvise(2) syscall allows to prioritize memory usage within a process.

Re: Modernizing Linux swapping: introducing the swap table

#104

Post by NoGravitas »

iberator wrote: post #3

Another useless feature into Linux kernel. Who uses swap space nowadays?! Last time I used swap on Linux device was around Pentium 2 era but in reality closer to 486DX era

Even if you have plenty of memory for your work load, there are useful performance reasons for having some swap. The TLDR section of this link covers the important bits.

https://chrisdown.name/2018/01/02/in-defence-of-swap.html

Relatively speaking, you do need a lot less swap than you did back in the day, but performance will suffer if you don't have some, and having too much doesn't cost you anything except storage.

Re: Modernizing Linux swapping: introducing the swap table

#105

Post by NoGravitas »

boomlinde wrote: post #54
man8alexd wrote: post #25

Earlier quoted context omitted.

The proper rule of thumb is to make the swap large enough to keep all inactive anonymous pages after the workload has stabilized, but not too large to cause swap thrashing and a delayed OOM kill if a fast memory leak happens.

That's not so much a rule of thumb as an assessment you can only make after thorough experimentation or careful analysis.

It doesn't take that much experimentation, though. Either set up not enough swap and keep increasing it by a little bit until you stop needing to increase it, or set up too much, and monitor your max use for a while (days/weeks), and then decrease it to a little more than the max you used.

Re: Modernizing Linux swapping: introducing the swap table

#106

Post by SAI_Peregrinus »

robinsonb5 wrote: post #75
direwolf20 wrote: post #62

Earlier quoted context omitted.

An Electron app would mark its entire 2GB as required for interactivity. If you run 4 electron apps on an 8GB system you run out of memory.

I don't mean interactivity within apps, per se - I mean the desktop and underlying OS, so that if an electron app goes unresponsive and eats all the free RAM the window manager can still kill it. Or you can still open a new terminal window, log in and kill it. Right now it can take several minutes to get a Linux system back under control once a swapstorm starts.

Linux doesn't really have any distinction between the desktop and underlying OS components in userspace and anything else in userspace. Linux is quite userland-agnostic, and distros have traditionally mixed user software with distro-managed software. You shouldn't use `sudo` to install software by default, your package manager should allow installing software for just your user. Software installed for the system could then be the only software allowed to mark itself as required for interactivity. You could do that manually to other software if you had root access, but "normal" user software installs with the package manager couldn't do so since they wouldn't get root access.

That'd require some new capabilities added, and some substantial shifts in how distro maintainers & users operate, so it's extremely unlikely. It's much closer to how things like Android operate, though still not quite as secure as giving each application its own user & dedicated storage for data.

Re: Modernizing Linux swapping: introducing the swap table

#107

Post by SAI_Peregrinus »

NoGravitas wrote: post #105
boomlinde wrote: post #54

Earlier quoted context omitted.

That's not so much a rule of thumb as an assessment you can only make after thorough experimentation or careful analysis.

It doesn't take that much experimentation, though. Either set up not enough swap and keep increasing it by a little bit until you stop needing to increase it, or set up too much, and monitor your max use for a while (days/weeks), and then decrease it to a little more than the max you used.

I went with "set up 0 swap" and then never needed to increase it. I built my PC in 2023, when RAM prices were still reasonable, stuck 128GiB of ECC DDR5 in, and haven't run into any need for swap. Start with 0, turn on zswap, and if you don't have enough RAM then make a swap file & set it up as backing for zswap.

Re: Modernizing Linux swapping: introducing the swap table

#108

Post by IshKebab »

NekkoDroid wrote: post #67
IshKebab wrote: post #48

Earlier quoted context omitted.

Yeah because Linux's memory management is quite poor and running out of RAM without swap will often mean a hard reboot. Swap definitely helps a lot, even if it doesn't fully solve the problem. To be honest I don't know why it's such an issue on Linux. Mac and Windows don't have this issue at all. Windows presumably because it doesn't over-commit memory. I'm not sure why Mac is so much better than Linux at memory mana…

> To be honest I don't know why it's such an issue on Linux. Mac and Windows don't have this issue at all. Windows presumably because it doesn't over-commit memory To be fair, my Windows system grinds to a halt (not really, but it becomes very noticably less responsive in basically anything) when JetBrains is installing an update (mind you I only have SSDs with all JetBrains stuff being on an NVMe). I don't know what…

I have had this happen in the past (not very often though), and another saving grave of Windows is you can press ctrl-alt-del, which somehow seems to pause the rest of the system activity, and then see a process list and choose which one to kill.

Linux doesn't have anything like that. KDE seems to have a somewhat functional Ctrl-alt-del menu - I have been able to access it when the rest of the shell gets screwed up (not due to OOM). But inexplicably the only options it has are Sleep, Restart, Shutdown or Log out!! Where is the "emergency shell", or "process manager" or even "run a program"? Ridiculous.

I think Linux GUIs often have this weird fetish with designing as if nothing will ever go wrong, which is clearly not how the real world works. Especially on Linux. I've genuinely heard people claim that most Linux users will never need to use a terminal for example.

Re: Modernizing Linux swapping: introducing the swap table

#109

Post by garaetjjte »

rustyhancock wrote: post #73
FooBarWidget wrote: post #2

One pet peeve I have with virtual memory management on Linux is that, as memory usage approaches 100%, the kernel starts evicting executable pages because technically they're read-only and can be loaded from disk. Thus, the entire system grinds to a halt in a behavior that looks like swapping, because every program that wants to execute instructions has to load its instructions from disk again, only to have those ins…

This explains some of the issues I was having on a laptop some months back. And searching desperately for "just kill the damn thing" option.

If you enable Magic SysRq you can invoke OOM-killer from keyboard. https://docs.kernel.org/admin-guide/sysrq.html
Post reply on HN