As hacks go, it's a good one. I also like it because you don't have to be root to implement it and you don't have to reconfigure your file system params in ways that might or might not be great for other reasons.
All my servers have an 8 GB empty file on disk
41–50 of 715 posts
Re: All my servers have an 8 GB empty file on disk
#42Ah, the classic 'speed-up loop' approach: https://thedailywtf.com/articles/The-Speedup-Loop About the blogpost itself: The disk filled up, and that's one thing you don't want on a Linux server—or a Mac for that matter. When the disk is full nothing good happens. I had this happen few times on a Mac and every time I was shocked that if disk gets full you cannot even delete a file and the only option is to do a full sy…
I don't know with Mac, but this is why many Linux distros recommend putting /home is on a separate partition. If it fills, it won't lock up the whole system.
Fun story with this. Ubuntu now has an experimental root-on-zfs feature. I installed it and started playing with some docker containers, trying to compile a certain version of pytorch. Suddenly, my computer crashed. Apparently, my root partition filled because docker installed everything on the same partition as my OS, crashing everything immediately.
Re: All my servers have an 8 GB empty file on disk
#43I think i know where this is going without even reading. Any attempt from the outside to pull this 8b
Gb file would be a very noticeable red flag.
Re: All my servers have an 8 GB empty file on disk
#44An alternative approach here... make sure (all) your filesystems are on top of LVM. This reduces the steps needed to grow your free space. Whether you have a 8gb empty file laying around, or an 8gb block device to attach...LVM will happily take them both as pv's, add them to your vg's, and finally expand your lv's.
some reading if LVM is new and you want to know more: https://opensource.com/business/16/9/linux-users-guide-lvm
edit to add: pv=physical volume, vg=volume group, lv=logical volume
Re: All my servers have an 8 GB empty file on disk
#45One other option is increasing the reserved block count ( https://ma.ttias.be/change-reserved-blocks-ext3-ext4-filesys... ). This has the nice side effect of increasing the space available for critical daemons. If you haven't customised this, in a pinch you can still lower it down a bit to buy some time.
I know about this, but I do think it's not a bad idea doing what he does because the reserved block count is for root and most server processes still run as root. And it's usually them that are causing the disk to fill. Though I suppose this also makes the problem itself more prominent in the first place. I guess if you run into this a lot, stricter monitoring would be a better solution. The way I found out about it…
What servers usually run as root? Some may start as root, but usually drop privileges for the actual server processes quickly, eg. apache, nginx, sshd.
Nothing that actually does the "serving" or accesses data should be running as root.
Re: All my servers have an 8 GB empty file on disk
#46I do similar, I keep multiple files though - 4GB, 2GB, 1GB and 100M, which I also use for testing speed
Re: All my servers have an 8 GB empty file on disk
#47One other option is increasing the reserved block count ( https://ma.ttias.be/change-reserved-blocks-ext3-ext4-filesys... ). This has the nice side effect of increasing the space available for critical daemons. If you haven't customised this, in a pinch you can still lower it down a bit to buy some time.
I suspect the blog author did not understand this (based on the content) - as a Linode user myself, I just had a look at one of my VMs and they install with the regular 5% reserved space (ext4/Debian).
Funny because I have always tune2fs -m1 or tune2fs -m0 because the reserved space was never supposed to scale linearly with hard drive capacities and is not useful to userspace in anyway. Have never had any issues and been doing it for decades in commercial applications. In some cases, where you probably shouldn't be using ext3/4 anyways, we are talking about reclaiming TBs of reserved space.
It's important to note that mkfs doesn't care if you are formatting the root partition or a data volume partition, it will still reserve space for the kernel.
Re: All my servers have an 8 GB empty file on disk
#48I'm sorry for meta comment, but this font is barely readable.
Not just the font of choice, but the formatting (hanging indent?). It makes it harder to read when there is little volume of text.
Re: All my servers have an 8 GB empty file on disk
#49This is clever. Our shaky version of this, historically, has been to run ncdu and hastily delete the first large log file we see. It’s not ideal.
ncdu saves my bacon at least every few months. I do machine learning and am always running out of space!
Re: All my servers have an 8 GB empty file on disk
#50I think i know where this is going without even reading. Any attempt from the outside to pull this 8b Gb file would be a very noticeable red flag.
Nope. Try again :)