Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Modernizing Linux swapping: introducing the swap table

lwn.net

Post reply on HN
Print view View original

81–90 of 132 posts

Re: Modernizing Linux swapping: introducing the swap table

#81

Post by FooBarWidget »

robinsonb5 wrote: post #4
FooBarWidget wrote: post #2

One pet peeve I have with virtual memory management on Linux is that, as memory usage approaches 100%, the kernel starts evicting executable pages because technically they're read-only and can be loaded from disk. Thus, the entire system grinds to a halt in a behavior that looks like swapping, because every program that wants to execute instructions has to load its instructions from disk again, only to have those ins…

Indeed. I think what's really needed is some way to mark pages as "required for interactivity" so that nothing related to the user interface gets paged out, ever. That, I think, would go at least some way towards restoring the feeling of "having a computer's full attention" that we had thirty years ago.

There is, mlock() or mlockall(), but it requires developer support. I wish there is an administrator knob that allows me to mark whole processes without needing to modify them.

Re: Modernizing Linux swapping: introducing the swap table

#82

Post by FooBarWidget »

man8alexd wrote: post #10
FooBarWidget wrote: post #2

One pet peeve I have with virtual memory management on Linux is that, as memory usage approaches 100%, the kernel starts evicting executable pages because technically they're read-only and can be loaded from disk. Thus, the entire system grinds to a halt in a behavior that looks like swapping, because every program that wants to execute instructions has to load its instructions from disk again, only to have those ins…

Actively used executable pages are explicitly excluded from reclaim. And if they are not used, why should they stay in memory when the memory is constrained? It is not the first time I have heard complaints about executable pages, but it seems to be some kind of common misunderstanding. https://news.ycombinator.com/item?id=45369516

What is "actively used"? The bash session that I was using 2 seconds before system grinded to a halt sure didn't count.

Re: Modernizing Linux swapping: introducing the swap table

#83

Post by man8alexd »

magicalhippo wrote: post #77
IshKebab wrote: post #48

Earlier quoted context omitted.

Yeah because Linux's memory management is quite poor and running out of RAM without swap will often mean a hard reboot. Swap definitely helps a lot, even if it doesn't fully solve the problem. To be honest I don't know why it's such an issue on Linux. Mac and Windows don't have this issue at all. Windows presumably because it doesn't over-commit memory. I'm not sure why Mac is so much better than Linux at memory mana…

> To be honest I don't know why it's such an issue on Linux. edit: I wrote all this before realizing I overlooked that you answered it yourself, so below is my very elaborate explanation of what you said: > Windows presumably because it doesn't over-commit memory. I'm no expert but from what I've gathered this ultimately boils down to how Linux went with fork for multiprocessing, vs Windows focused on threads. With f…

> However in almost all normal cases, this grossly overestimates the required memory and thus leads to swapping when technically it is not needed.

This is not true. Disabling overcommit doesn't change reclaim and swapping behaviour and doesn't lead to unnecessary swapping.

Re: Modernizing Linux swapping: introducing the swap table

#84

Post by 112233 »

direwolf20 wrote: post #63
112233 wrote: post #24

Earlier quoted context omitted.

Is there a way to make linux kernel schedule in a "batch friendly way"? Say I do "make -j" and get 200 gcc processes diong jobserver LTO link with 2GB RSS each. In my head, optimal way through such mess is get as many processes as can fit into RAM without swapping, run them to completion, and schedule additional processes as resources become available. A depth first, "infinite latency" mode. Any combination of cgroup…

It's not possible for the kernel to predict the memory needs of a process unfortunately

But how about not scheduling swapped out processes if there currently is no free ram for their current RSS? of course kernel cannot know that a new process will balloon to eat all RAM, but once it has done so, is there a way to let it run to completion without being swapped out to "improve responsivity"?

Re: Modernizing Linux swapping: introducing the swap table

#85

Post by man8alexd »

FooBarWidget wrote: post #82
man8alexd wrote: post #10

Earlier quoted context omitted.

Actively used executable pages are explicitly excluded from reclaim. And if they are not used, why should they stay in memory when the memory is constrained? It is not the first time I have heard complaints about executable pages, but it seems to be some kind of common misunderstanding. https://news.ycombinator.com/item?id=45369516

What is "actively used"? The bash session that I was using 2 seconds before system grinded to a halt sure didn't count.

Your bash session is most likely still in memory but the system is spending 99.9% of time waiting for swap I/O, trying to free a few pages of memory. Swap random access latency is 10^3 slower than RAM.

Re: Modernizing Linux swapping: introducing the swap table

#86

Post by 112233 »

Neywiny wrote: post #76
112233 wrote: post #24

Earlier quoted context omitted.

Is there a way to make linux kernel schedule in a "batch friendly way"? Say I do "make -j" and get 200 gcc processes diong jobserver LTO link with 2GB RSS each. In my head, optimal way through such mess is get as many processes as can fit into RAM without swapping, run them to completion, and schedule additional processes as resources become available. A depth first, "infinite latency" mode. Any combination of cgroup…

"make -j" has OOMd me more than it's worth. If it's a big project I just put in how many threads I want. I do hear your point but that is a solved problem.

actually, global jobserver is another unsolved thing that seems unvelievable nobody has done yet. You have server. Server spins N containers (kubes, dockers, multiple user sessions ...), each of them is building something. There is no mechanism to run batch of tasks in parallel in a way that uses available cores. Some special cases (make/ninja/gcc) work, but no general mechanism I know of

Re: Modernizing Linux swapping: introducing the swap table

#87

Post by tremon »

112233 wrote: post #44
tremon wrote: post #30

Earlier quoted context omitted.

I'm also in the GP's camp; RAM is for volatile data, disk is for data persistence. The first "why would you do that" that needs to be addressed is why volatile data should be written to disk. And "it's just a few % of your disk" is not a sufficient answer to that question.

> RAM is for volatile data, disk is for data persistence. Genuinely curious where this idea has come from. Is it something being taught currently?

No, not currently -- since the start of computers. This is quite literally part of Computing 101; see https://web.stanford.edu/class/cs101/lecture02.html#/9 , slides 10-12.

You can ask your favourite search engine or language fabricator about the differences between RAM and disk storage, they will all tell you the same thing. Frankly, it's kind of astonishing that this needs to be explained on a site like HN.

Re: Modernizing Linux swapping: introducing the swap table

#88

Post by [deleted] »

homebrewer wrote: post #36
ChocolateGod wrote: post #26

I'd like to see Linux gain support for actual memory compression, without the need to go through zram, similar to macOS/Windows.

zram has been "obsolete" for years, I don't know why people still reach for it. Linux supports proper memory compression in the form of zswap https://wiki.archlinux.org/title/Zswap

[deleted]

Re: Modernizing Linux swapping: introducing the swap table

#89

Post by [deleted] »

homebrewer wrote: post #36
ChocolateGod wrote: post #26

I'd like to see Linux gain support for actual memory compression, without the need to go through zram, similar to macOS/Windows.

zram has been "obsolete" for years, I don't know why people still reach for it. Linux supports proper memory compression in the form of zswap https://wiki.archlinux.org/title/Zswap

[deleted]

Re: Modernizing Linux swapping: introducing the swap table

#90

Post by rascul »

ZoomZoomZoom wrote: post #60
man8alexd wrote: post #42

Earlier quoted context omitted.

You are right, it is not a rule of thumb, and you can't determine optimal swap size right away. But you don't need "extensive runtime analysis". Start with a small swap - a few hundred megabytes (assuming the system has GBs of RAM). Check its utilization periodically. If it is full, add a few hundred megabytes more. That's all.

It's not like it's easy to shuffle partitions around. Swap files are a pain, so you need to reserve space at the end of the table. By the time you need to increase swap the previous partition is going to be full. Better overcommit right away and live with the feeling you're wasting space.

> Swap files are a pain

Easier than partitions:

    mkswap --size 2G --file swap.img
    swapon swap.img
Post reply on HN