Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

All my servers have an 8 GB empty file on disk

brianschrader.com

Post reply on HN
Print view View original

61–70 of 715 posts

Re: All my servers have an 8 GB empty file on disk

#61

Post by LinuxBender »

jstanley wrote: post #10
00deadbeef wrote: post #7

Isn’t using LVM and holding some space back a better solution for this? Also I keep databases on their own partition so that nothing else can accidentally fill up the space and lead to data loss.

Maybe, but author says: > On Linux servers it can be incredibly difficult for any process to succeed if the disk is full. You won't feel too clever if you come to grow your LVM volume into the free space and it won't work because there's no free space on the filesystem! :) (I don't actually know if this would fail or not - but the point is "rm spacer.img" is pretty much guaranteed not to fail).

I've used LVM for this purpose plenty of times. lvm2 at least has not prevented me from extending a full disk. lvm + reserved blocks + a small spacer file are all decent options, even better when used together.

Re: All my servers have an 8 GB empty file on disk

#62

Post by Deathmax »

Denvercoder9 wrote: post #9

> Copy commands and even deletions can fail or take forever as memory tries to swap to a full disk That's only a problem if your memory is full as well, and even then, I've never encountered a server that uses a swapfile instead of a swap partition.

Not quite the same situation as described in the article, but it is still possible for the kernel to swap memory in and out of disk even without a swap file/partition. Memory used for storing executable binaries is allowed to be moved out of memory, as a copy of it lives on disk. This means you can still encounter memory thrashing (and thus system unresponsiveness) under low memory situations.

Re: All my servers have an 8 GB empty file on disk

#64

Post by ttyprintk »

Something1234 wrote: post #19

I have an empty leader on my hard drive so that I can recover if I accidentally nuke the front of it with dd while making a live usb. So it's not a bad idea, and it's super effective so far it hasn't been tested, and hopefully I never will need to.

A good reason to partition swap before /boot.

Re: All my servers have an 8 GB empty file on disk

#65

Post by scottlamb »

> On Linux servers it can be incredibly difficult for any process to succeed if the disk is full. Copy commands and even deletions can fail or take forever as memory tries to swap to a full disk and there's very little you can do to free up large chunks of space.

This reasoning doesn't make sense. On Linux, swap is preallocated. This is true regardless of whether you're using a swap partition or a swap file. See man swapon(8):

> The swap file implementation in the kernel expects to be able to write to the file directly, without the assistance of the filesystem. This is a problem on files with holes or on copy-on-write files on filesystems like Btrfs.

> Commands like cp(1) or truncate(1) create files with holes. These files will be rejected by swapon.

I just verified on Linux 5.8.0-48-generic (Ubuntu 20.10) / ext4 that trying to swapon a sparse file fails with "skipping - it appears to have holes".

Now, swap is horribly slow, particularly on spinning rust rather than SSD. I run my systems without any swap for that reason. But swapping shouldn't fail on a full filesystem, unless you're trying to create & swapon a new swapfile after the filesystem is filled.

Re: All my servers have an 8 GB empty file on disk

#67

Post by ttyprintk »

1123581321 wrote: post #6

This is clever. Our shaky version of this, historically, has been to run ncdu and hastily delete the first large log file we see. It’s not ideal.

The filesystems where headroom matters are var, tmp and sometimes root. I like this strategy with logfiles because nethack.log.gz.30 was approximately as important as empty space. Keeping another 8gb on root and tmp seems extreme.

Re: All my servers have an 8 GB empty file on disk

#69

Post by michelb »

Showing off that i’m not a sysadmin, but wouldn’t a monitoring daemon work? Once disk usage grows past a certain uncomfortable threshold you get an email/notification to see what’s up. I mean you obviously are monitoring other server vitals anyway right?
Post reply on HN