Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

All my servers have an 8 GB empty file on disk

brianschrader.com

Post reply on HN
Print view View original

21–30 of 715 posts

Re: All my servers have an 8 GB empty file on disk

#21

Post by Spivak »

00deadbeef wrote: post #7

Isn’t using LVM and holding some space back a better solution for this? Also I keep databases on their own partition so that nothing else can accidentally fill up the space and lead to data loss.

Tomato potato. If you use LVM or anything like it to reserve space then in your failure situation you have to extend the lv, partition, and fs before the space becomes available. More work than just rm'ing the file.

I think the ideal ideal is tuning the reserved blocks in your filesystem.

   xfs_io -x -c 'resblks ...

Re: All my servers have an 8 GB empty file on disk

#22

Post by JshWright »

00deadbeef wrote: post #17
JshWright wrote: post #12

Earlier quoted context omitted.

rm /path/to/big.file is faster than looking up the commands to expand the LVM volume and grow the filesystem.

They’re not hard to memorise. I never look them up.

Good for you. That's not how my memory works. If I don't use a command regularly, I don't trust myself to remember it correctly. Even if I did though, that's a multi-step process, compared to the single command needed to remove a file.

Re: All my servers have an 8 GB empty file on disk

#23

Post by ed25519FUUU »

loloquwowndueo wrote: post #14

If you don’t have monitoring to tell you when the disk is more than X% full, then you’re at risk for more failure scenarios than just a full disk (usually trivial to buy time by deleting old logs).

Happens all of the time even with monitoring. Somebody enables debug monitoring and it fills up in 3 minutes.

Re: All my servers have an 8 GB empty file on disk

#24

Post by stonesweep »

AlisdairO wrote: post #11

One other option is increasing the reserved block count ( https://ma.ttias.be/change-reserved-blocks-ext3-ext4-filesys... ). This has the nice side effect of increasing the space available for critical daemons. If you haven't customised this, in a pinch you can still lower it down a bit to buy some time.

I suspect the blog author did not understand this (based on the content) - as a Linode user myself, I just had a look at one of my VMs and they install with the regular 5% reserved space (ext4/Debian).

Re: All my servers have an 8 GB empty file on disk

#25

Post by terramex »

Ah, the classic 'speed-up loop' approach: https://thedailywtf.com/articles/The-Speedup-Loop

About the blogpost itself:

The disk filled up, and that's one thing you don't want on a Linux server—or a Mac for that matter. When the disk is full nothing good happens.

I had this happen few times on a Mac and every time I was shocked that if disk gets full you cannot even delete a file and the only option is to do a full system reboot. I was also unable to save any open file, even to external disk and suffered minor data loss every time due to that.

What is the proper way of dealing with such issue on macOS? (or other systems, if they behave the same way)

Re: All my servers have an 8 GB empty file on disk

#26

Post by yjftsjthsd-h »

secabeen wrote: post #5
rektide wrote: post #3

hope you're not running -o compress=lz4 , because you are going to be in for a big surprise when you try to pull this emergency lever! you may be shocked to see you don't actually get much space back! i do wonder how many FS would actually allocate the 8GB if you, for example, opened a file, seeked to 8GB mark, and wrote a character. many file systems support "sparse files"[1]. for example on btrfs, i can run 'dd if=…

> hope you're not running -o compress=lz4 , because you are going to be in for a big surprise when you try to pull this emergency lever! you may be shocked to see you don't actually get much space back! This is true. If you are replicating this, copy from /dev/urandom rather than using an empty file.

It feels more elegant to do something like `touch /bigfile && chattr -c /bigfile && truncate --size 8G` to outright disable compression on that file

Re: All my servers have an 8 GB empty file on disk

#27

Post by 00deadbeef »

Spivak wrote: post #21
00deadbeef wrote: post #7

Isn’t using LVM and holding some space back a better solution for this? Also I keep databases on their own partition so that nothing else can accidentally fill up the space and lead to data loss.

Tomato potato. If you use LVM or anything like it to reserve space then in your failure situation you have to extend the lv, partition, and fs before the space becomes available. More work than just rm'ing the file. I think the ideal ideal is tuning the reserved blocks in your filesystem. xfs_io -x -c 'resblks ...

It’s not really much more work. One command extra.

Re: All my servers have an 8 GB empty file on disk

#29

Post by sonicrocketman »

AlisdairO wrote: post #11

One other option is increasing the reserved block count ( https://ma.ttias.be/change-reserved-blocks-ext3-ext4-filesys... ). This has the nice side effect of increasing the space available for critical daemons. If you haven't customised this, in a pinch you can still lower it down a bit to buy some time.

I did not know this. Good solution.
Post reply on HN