Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Modernizing Linux swapping: introducing the swap table

lwn.net

Post reply on HN
Print view View original

61–70 of 132 posts

Re: Modernizing Linux swapping: introducing the swap table

#61

Post by man8alexd »

ZoomZoomZoom wrote: post #60
man8alexd wrote: post #42

Earlier quoted context omitted.

You are right, it is not a rule of thumb, and you can't determine optimal swap size right away. But you don't need "extensive runtime analysis". Start with a small swap - a few hundred megabytes (assuming the system has GBs of RAM). Check its utilization periodically. If it is full, add a few hundred megabytes more. That's all.

It's not like it's easy to shuffle partitions around. Swap files are a pain, so you need to reserve space at the end of the table. By the time you need to increase swap the previous partition is going to be full. Better overcommit right away and live with the feeling you're wasting space.

Exactly opposite. Don't use swap partitions, and use swap files, even multiple if necessary. Never allocate too much swap space. It is better to get OOM earlier then to wait for unresponsive system.

Re: Modernizing Linux swapping: introducing the swap table

#62

Post by direwolf20 »

robinsonb5 wrote: post #4
FooBarWidget wrote: post #2

One pet peeve I have with virtual memory management on Linux is that, as memory usage approaches 100%, the kernel starts evicting executable pages because technically they're read-only and can be loaded from disk. Thus, the entire system grinds to a halt in a behavior that looks like swapping, because every program that wants to execute instructions has to load its instructions from disk again, only to have those ins…

Indeed. I think what's really needed is some way to mark pages as "required for interactivity" so that nothing related to the user interface gets paged out, ever. That, I think, would go at least some way towards restoring the feeling of "having a computer's full attention" that we had thirty years ago.

An Electron app would mark its entire 2GB as required for interactivity. If you run 4 electron apps on an 8GB system you run out of memory.

Re: Modernizing Linux swapping: introducing the swap table

#63

Post by direwolf20 »

112233 wrote: post #24
FooBarWidget wrote: post #2

One pet peeve I have with virtual memory management on Linux is that, as memory usage approaches 100%, the kernel starts evicting executable pages because technically they're read-only and can be loaded from disk. Thus, the entire system grinds to a halt in a behavior that looks like swapping, because every program that wants to execute instructions has to load its instructions from disk again, only to have those ins…

Is there a way to make linux kernel schedule in a "batch friendly way"? Say I do "make -j" and get 200 gcc processes diong jobserver LTO link with 2GB RSS each. In my head, optimal way through such mess is get as many processes as can fit into RAM without swapping, run them to completion, and schedule additional processes as resources become available. A depth first, "infinite latency" mode. Any combination of cgroup…

It's not possible for the kernel to predict the memory needs of a process unfortunately

Re: Modernizing Linux swapping: introducing the swap table

#64

Post by direwolf20 »

IshKebab wrote: post #48
dist-epoch wrote: post #31

Both Canonical and Microsoft recommend enabling swap file for Ubuntu cloud images, even if you allocate plenty of RAM to the VM. Any thoughts on that?

Yeah because Linux's memory management is quite poor and running out of RAM without swap will often mean a hard reboot. Swap definitely helps a lot, even if it doesn't fully solve the problem. To be honest I don't know why it's such an issue on Linux. Mac and Windows don't have this issue at all. Windows presumably because it doesn't over-commit memory. I'm not sure why Mac is so much better than Linux at memory mana…

My experience is different. Running out of RAM without swap will cause the most memory–hungry process to die, whereupon systemd restarts it. Running out of RAM with swap causes thrashing and you can't serve any requests or ssh logins. Someone has to press the reset button then.

Re: Modernizing Linux swapping: introducing the swap table

#65

Post by direwolf20 »

GCUMstlyHarmls wrote: post #13
ch_123 wrote: post #7

Earlier quoted context omitted.

I ran Linux without swap for some years on a laptop with a large-for-the-time amount of RAM (about 8GB). It _mostly_ worked, but sudden spikes of memory usage would render the system unresponsive. Usually it would recover, but it in some cases it required a power cycle. Similarly, on a server where you might expect most of the physical memory to get used, it ends up being very important for stability. Think of VM or…

I dont get why anti-swap is so prevalent in Linux discussions. Like, what does it hurt to stick 8-16-32gb extra "oh fuck" space on your drive. Either you're going to never exhaust your system ram, so it doesn't matter, minimally exhaust it and swap in some peak load but at least nothing goes down, or exhaust it all and start having things get OOM'd which feels bad to me. Am I out of touch? Surely it's the children wh…

Swap causes thrashing, making the whole system unusable, instead of a clean OOM kill

Re: Modernizing Linux swapping: introducing the swap table

#66

Post by direwolf20 »

ZoomZoomZoom wrote: post #60
man8alexd wrote: post #42

Earlier quoted context omitted.

You are right, it is not a rule of thumb, and you can't determine optimal swap size right away. But you don't need "extensive runtime analysis". Start with a small swap - a few hundred megabytes (assuming the system has GBs of RAM). Check its utilization periodically. If it is full, add a few hundred megabytes more. That's all.

It's not like it's easy to shuffle partitions around. Swap files are a pain, so you need to reserve space at the end of the table. By the time you need to increase swap the previous partition is going to be full. Better overcommit right away and live with the feeling you're wasting space.

Hast thou discovered our lord and savior LVM?

Re: Modernizing Linux swapping: introducing the swap table

#67

Post by NekkoDroid »

IshKebab wrote: post #48
dist-epoch wrote: post #31

Both Canonical and Microsoft recommend enabling swap file for Ubuntu cloud images, even if you allocate plenty of RAM to the VM. Any thoughts on that?

Yeah because Linux's memory management is quite poor and running out of RAM without swap will often mean a hard reboot. Swap definitely helps a lot, even if it doesn't fully solve the problem. To be honest I don't know why it's such an issue on Linux. Mac and Windows don't have this issue at all. Windows presumably because it doesn't over-commit memory. I'm not sure why Mac is so much better than Linux at memory mana…

> To be honest I don't know why it's such an issue on Linux. Mac and Windows don't have this issue at all. Windows presumably because it doesn't over-commit memory

To be fair, my Windows system grinds to a halt (not really, but it becomes very noticably less responsive in basically anything) when JetBrains is installing an update (mind you I only have SSDs with all JetBrains stuff being on an NVMe). I don't know what JetBrains is doing, but it consistently makes itself noticable when it is updating.

Re: Modernizing Linux swapping: introducing the swap table

#68

Post by man8alexd »

direwolf20 wrote: post #63
112233 wrote: post #24

Earlier quoted context omitted.

Is there a way to make linux kernel schedule in a "batch friendly way"? Say I do "make -j" and get 200 gcc processes diong jobserver LTO link with 2GB RSS each. In my head, optimal way through such mess is get as many processes as can fit into RAM without swapping, run them to completion, and schedule additional processes as resources become available. A depth first, "infinite latency" mode. Any combination of cgroup…

It's not possible for the kernel to predict the memory needs of a process unfortunately

It is possible to measure process memory utilitsation and set appropriate cgroup limits.

Re: Modernizing Linux swapping: introducing the swap table

#69

Post by man8alexd »

direwolf20 wrote: post #65
GCUMstlyHarmls wrote: post #13

Earlier quoted context omitted.

I dont get why anti-swap is so prevalent in Linux discussions. Like, what does it hurt to stick 8-16-32gb extra "oh fuck" space on your drive. Either you're going to never exhaust your system ram, so it doesn't matter, minimally exhaust it and swap in some peak load but at least nothing goes down, or exhaust it all and start having things get OOM'd which feels bad to me. Am I out of touch? Surely it's the children wh…

Swap causes thrashing, making the whole system unusable, instead of a clean OOM kill

swap causes thrashing if you have too large swap and no cgroup limits.
Post reply on HN