Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

A deep dive into Debian 13 /tmp: What's new, and what to do if you don't like it

lowendbox.com

Post reply on HN
Print view View original

71–80 of 137 posts

Re: A deep dive into Debian 13 /tmp: What's new, and what to do if you don't like it

#71

Post by mnw21cam »

gnyman wrote: post #18
buckle8017 wrote: post #14

Earlier quoted context omitted.

Which is a great reason to have a big swap file now.

Note though that if you don't have swap now, and enable it, you introduce the risk of thrashing [1] If you have swap already it doesn't matter, but I've encountered enough thrashing that I now disable swap on almost all servers I work with. It's rare but when it happens the server usually becomes completely unresponsive, so you have to hard reset it. I'd rather that the application trying to use too much memory is ki…

That's not true. Without swap, you already have the risk of thrashing. This is because Linux views all segments of code which your processes are running as clean and evictable from the cache, and therefore basically equivalent to swap, even when you have no swap. Under low-memory conditions, Linux will happily evict all clean pages, including the ones that the next process to be scheduled needs to execute from, causing thrashing. You can still get an unresponsive server under low memory conditions due to thrashing with no swap.

Setting swappiness to zero doesn't fix this. Disabling swap doesn't fix this. Disabling overcommit does fix this, but that might have unacceptable disadvantages if some of the processes you are running allocate much more RAM than they use. Installing earlyoom to prevent real low memory conditions does fix this, and is probably the best solution.

Re: A deep dive into Debian 13 /tmp: What's new, and what to do if you don't like it

#72

Post by cromka »

yjftsjthsd-h wrote: post #42
abrookewood wrote: post #27

Earlier quoted context omitted.

That's a really good point that had never occurred to me. Edit: I think that the use of ZFS for your /tmp would solve this. You get Error Corrected memory writing to an check-summed file system.

ZFS /tmp is probably fine, but swapping to ZFS on Linux is dicey AIUI; there's an unfortunate possibility of deadlock https://github.com/openzfs/zfs/issues/7734

So maybe another filesystem with heavy checksums could be used? Btrfs or dm-crypt with integrity over ext4?

Re: A deep dive into Debian 13 /tmp: What's new, and what to do if you don't like it

#73

Post by mnw21cam »

cwillu wrote: post #15
saurik wrote: post #12

Earlier quoted context omitted.

Why is reading the data back from swap be slower at all -- much less "DRAMATICALLY" so -- than saving the data to disk and reading it back?

Because swapping back in happens 4kb at a time

It's also because a filesystem is much more likely to have consecutive parts of a file stored consecutively on disc, whereas swap is going to just randomly scatter 4kB blocks everywhere, so you'll be dealing with random access read speed instead of throughput read speed.

Re: A deep dive into Debian 13 /tmp: What's new, and what to do if you don't like it

#74

Post by Mashimo »

wielebny wrote: post #32
perlgeek wrote: post #22

Earlier quoted context omitted.

Who runs around with a 100gb+ /tmp partition? Our default server images come with a 4.4GB /tmp partition...

I run a script that rotates my /tmp/ each day, so I can access yesterday's tmp files at /tmp/20250828/ and so on. My /tmp is my default folder for downloads and temporary work. It will grow 100GB+ easily.

For a long time my default download folder was /dev/shm. It is / was? the memory tmpfs and everything would just be gone after a reboot. Now I can just use /tmp

Even used something similar on my windows pc, had a B:/ disk 1GB in size that was my download folder. Automated cleanup made easy.

Re: A deep dive into Debian 13 /tmp: What's new, and what to do if you don't like it

#75

Post by mnw21cam »

mhitza wrote: post #29
magicalhippo wrote: post #16

Earlier quoted context omitted.

If you're running it in a VM you might not have all that luxurious RAM. When my Linux VM starts swapping I have to either wait an hour or more to regain control, or just hard restart the VM.

What distro are you running? systemd-oomd kills processes a bit quicker than what came before (a couple minutes of a slow, stuttery system). Still too slow for a server you'd want to have back online as quickly as possible. At least now when I run out of memory it kills processes that consume the most memory. A few years back it used to kill my desktop session instead!

Right, that's traditionally been because the X server has typically had a fairly large footprint, and therefore has been very attractive for the oom killer. But in the last 15 years or so, some heuristics have been applied to deliberately discourage the oom killer from killing "important things".

I install earlyoom on systems I admin. It prevents the low-memory thrashing by killing things while the system is still responsive, instead of when the system is in a state that means it'll take hours to recover.

Re: A deep dive into Debian 13 /tmp: What's new, and what to do if you don't like it

#76

Post by loa_in_ »

cl3misch wrote: post #66
loa_in_ wrote: post #54

Earlier quoted context omitted.

I tried out variations on this on my daily driver setups. The design choices here were likely threefold: Store tmpfs in memory: volatile but limited to free ram or swap, and that writes to disk Store tmpfs on dedicated volume: Since we're going to write onto disk anyway, make it a lightweight special purpose file system that's commited to disk On disk tmpfs but cleaned up periodically: additional settings to clean up…

A tmpfs itself is basically a ramdisk by definition. I assume you mean /tmp when you say tmpfs?

Yes. I'm not careful lately.

Re: A deep dive into Debian 13 /tmp: What's new, and what to do if you don't like it

#77

Post by tatref »

cromka wrote: post #72
yjftsjthsd-h wrote: post #42

Earlier quoted context omitted.

ZFS /tmp is probably fine, but swapping to ZFS on Linux is dicey AIUI; there's an unfortunate possibility of deadlock https://github.com/openzfs/zfs/issues/7734

So maybe another filesystem with heavy checksums could be used? Btrfs or dm-crypt with integrity over ext4?

Why not dm-integrity?

Re: A deep dive into Debian 13 /tmp: What's new, and what to do if you don't like it

#78

Post by snmx999 »

mrweasel wrote: post #59
greatgib wrote: post #49

I'm surprised to discover that it was not already the case for a long time for tmpfs to be used for /tmp, and that change is nice. But the auto-cleanup feature looks awful to me. Be it desktop or servers, machine with uptime of more than a year, I never saw the case of tmp being filled just by forgotten garbage. Only sometimes filled by unzipping a too big file or something like that. But it is on the spot. It used t…

> I never saw the case of tmp being filled just by forgotten garbage. It might have more to do with the type of developers I've worked with, but it happens all the time. Monitoring complains and you go into check, and there it is gigabytes of junk dumped there by shitty software or scripts that can't cleanup after themselves. The issue is that you don't always knows what's safe to delete, if you're the operations per…

Autocleaning: get the last accessed time from a file and only auto-clean files not accessed in the last n hours, e.g. 24 hours? Should be reasonably safe.

Re: A deep dive into Debian 13 /tmp: What's new, and what to do if you don't like it

#79

Post by marginalia_nu »

mnw21cam wrote: post #69
marginalia_nu wrote: post #56

Earlier quoted context omitted.

The Linux OOM killer is kinda sketchy to rely on. It likes to freeze up your system for long periods of time as it works out how to resolve the issue. Then it starts killing random PIDs to try to reclaim RAM like a system wide russian roulette. It's especially janky when you don't have swap. I've found adding a small swap file of ~500 MB makes it work so much better, even for systems with half a terabyte of RAM this…

Install earlyoom or one of its near-equivalents. That mostly solves the problem of it freezing up the system for long periods of time. I haven't personally seen the OOM killer kill unproductively - usually it kills either a runaway culprit or something that will actually free up enough space to help. For your "even for systems with half a terabyte of RAM", it is logical that the larger the system, the worse this beha…

> For your "even for systems with half a terabyte of RAM", it is logical that the larger the system, the worse this behaviour is, because when things go sideways there is a lot more stuff to sort out and that takes longer. My work server has 1.5TB of RAM, and an OOM event before I installed earlyoom was not pretty at all.

I meant it more in the sense that it doesn't have to be more than a few hundred MB even for large RAM. It's not the size of the swap file that makes the difference, but its presence, and advice of having it be proportional to RAM are largely outdated.

Re: A deep dive into Debian 13 /tmp: What's new, and what to do if you don't like it

#80

Post by baq »

worthless-trash wrote: post #37
baq wrote: post #24

Earlier quoted context omitted.

This is why I’m running with overcommit 2 and a different ratio per server purpose. …though I’m not sure why we have to think about this in 2025 at all.

I'm assuming that you monitor the service closely for OOM then adjust with demand ?

yeah pretty much, also configuring memory limits everywhere where apps allow it. some software also handles malloc failures relatively gracefully, which helps a whole lot (thank you postgres devs)
Post reply on HN