All my servers have an 8 GB empty file on disk • Hacker News Board

Re: All my servers have an 8 GB empty file on disk

#711

Post by MayeulC » Mon, Mar 29, 2021, 6:10 PM UTC

scottlamb wrote: post #346

MayeulC wrote: post #250

Earlier quoted context omitted.

IIRC, swap is actually needed for some memory operations. And when you run out of memory, the behaviour is often worse without swap. These days I always at least configure an in-memory compressed swap (zram).

You recall incorrectly; swap is not needed. It's not just me who runs without it; Google production machines did for many years. "The behavior is often worse without swap" is more vague / subjective. I prefer that a process die cleanly than everything slow to a crawl semi-permanently. I've previously written about swap causing the latter: https://news.ycombinator.com/item?id=13715917 To some extent the bad behavior c…

Well, I'm not inventing it. This phoronix article [1] leads to a LKML post [2]:

> Yeah that's a known problem, made worse SSD's in fact, as they are able > to keep refaulting the last remaining file pages fast enough, so there > is still apparent progress in reclaim and OOM doesn't kick in.

And so on. That's part of the reason there is a working effort towards creating a userspace OOM daemon. I'm grateful for that effort, since I'd rather have some apps crash than having my system unusable. However, there is an issue with the Linux kernel trashing.

I couldn't find mention of it in that thread, but it's mentioned multiple times in passing (as well as in the phoronix comments) that the system behaves worse without swap. It's also noted as one of the reproduction steps.

I agree that it's vague and subjective, but I'm pretty sure I've seen some serious discussion of the topic. There are a few sources that point in that direction [3-4], but I haven't seen anything conclusive.

[1]: https://www.phoronix.com/scan.php?page=news_item&px=Linux-Do...

[2]: https://lkml.org/lkml/2019/8/5/416

[3]: https://unix.stackexchange.com/a/28176

[4]: https://serverfault.com/questions/255661/linux-oom-disk-i-o-...

Re: All my servers have an 8 GB empty file on disk

#712

Post by MayeulC » Mon, Mar 29, 2021, 6:21 PM UTC

zamadatix wrote: post #314

MayeulC wrote: post #250

Earlier quoted context omitted.

IIRC, swap is actually needed for some memory operations. And when you run out of memory, the behaviour is often worse without swap. These days I always at least configure an in-memory compressed swap (zram).

Depends if you want things to gracefully degrade because you know you don't have enough RAM or if you'd rather things just straight up die. E.g. for the things I work on my laptop with if whatever I do isn't going to work with 128 GB of RAM (80% of which was meant to be cached data not actually used) then it's because it went horribly wrong and needs to be halted not because I needed some swap which is just going to…

Personally, I prefer things that use a huge amount of memory to die instead of having the machine become unresponsive, especially if I have latency-sensitive things at the same time, like a conference call.

Now, my main issue is that disabling swap doesn't cause this. With swap, OOM ends up being detected and corrected by the kernel. On some machines, it doesn't happen without swap, and often neither with swap. A sweet spot seems to be current-to-last CPU gen with 8 GB of memory.

There's probably a lot of factors at play: what makes a task "hung"? How memory pressure is measured. Stuff probably changed with time, SSDs are now pretty quick, yet storage is a lot slower than RAM compared to what it used to be.

Anyway, I use zram and have upgraded my memory, I'll also increase a bit my swap partition size in light of all of this. I'll probably have a look at userspace OOM daemons too.

Re: All my servers have an 8 GB empty file on disk

#713

Post by Darkphibre » Tue, Mar 30, 2021, 7:49 AM UTC

tuzakey wrote: post #360

jedberg wrote: post #309

Earlier quoted context omitted.

My friend had a truck with a reserve tank, but it was the same size as the main tank, so he would just flip the switch at every fill up to make sure they both got used.

This is typically used for agricultural/off-road fuel which is not priced with road taxes and as a result much cheaper. Off road fuel is dyed red in the US. If you get caught running dyed diesel on road you will be fined. Thus the switch on the dash, when you leave the highway to drive on your farm you flip over to dyed fuel to save $$.

Oh, fascinating! My first vehicle was the family's 3/4-ton Diesel '84 Chevy Pickup from the farm, and I'd forgotten it had an Aux fuel tank! This makes a lot of sense.

Re: All my servers have an 8 GB empty file on disk

#714

Post by hollerith » Tue, Mar 30, 2021, 6:16 PM UTC

1970-01-01 wrote: post #494

Tell all your SecDevOps friends how this file can also pull double duty as a ransomware canary. https://blog.urbackup.org/371/ransomware-canary https://support.huntress.io/article/136-ransomware-canaries-...

OK, but any human-written text file can serve as a ransomware canary because it is straightforward for code to distinguish between plaintext written by a human and encrypted text.

Re: All my servers have an 8 GB empty file on disk

#715

Post by ianmcgowan » Fri, Apr 02, 2021, 5:06 PM UTC

realaether wrote: post #703

ianmcgowan wrote: post #557

Earlier quoted context omitted.

Or for the more technical folk with access to a linux server, setup postfix/dovecot, connect outlook to it and arrange for archived emails to go to the IMAP server. The IT people get smart about looking for OST or PST files, but let's see them catch that :-)

Any pointers on getting started with the postfix/dovecot method?

If you have a Linux box (or a VM), install and configure it to route mail and provide IMAP support. Digital Ocean has the best tutorials for this:

https://www.digitalocean.com/community/tutorials/how-to-set-...

Then configure a new mail account in Outlook and connect to the IMAP server. It's optional, but convenient for replies, to configure the account to send via postfix if you have an internal SMTP server to connect to.

I gave up on email folders years ago, so at the end of the month would just create two new folders in the archive account (YYYYMM and YYYYMM_Sent) and drag all the mail from the Exchange account into the IMAP folders. Et voila! You now have your own local email archive.