Live data from Hacker News

All my servers have an 8 GB empty file on disk

brianschrader.com

31–40 of 715 posts

Re: All my servers have an 8 GB empty file on disk

#31
Same idea as this game development legend

https://www.dodgycoder.net/2012/02/coding-tricks-of-game-dev...

> he had put aside those two megabytes of memory early in the development cycle. He knew from experience that it was always impossible to cut content down to memory budgets, and that many projects had come close to failing because of it. So now, as a regular practice, he always put aside a nice block of memory to free up when it's really needed.

Re: All my servers have an 8 GB empty file on disk

#34

If you don’t have monitoring to tell you when the disk is more than X% full, then you’re at risk for more failure scenarios than just a full disk (usually trivial to buy time by deleting old logs).

If the problem arises during a migration or other significant event, which it sounds like Marco's did, the alert will usually be triggered just in time to tell you why you're already in a world of pain.

Re: All my servers have an 8 GB empty file on disk

#35

One other option is increasing the reserved block count ( https://ma.ttias.be/change-reserved-blocks-ext3-ext4-filesys... ). This has the nice side effect of increasing the space available for critical daemons. If you haven't customised this, in a pinch you can still lower it down a bit to buy some time.

I know about this, but I do think it's not a bad idea doing what he does because the reserved block count is for root and most server processes still run as root. And it's usually them that are causing the disk to fill. Though I suppose this also makes the problem itself more prominent in the first place. I guess if you run into this a lot, stricter monitoring would be a better solution.

The way I found out about it originally was because I was using external storage drives and I was never able to fit as much as I expected :D

Luckily you can easily change this without reformatting.

Re: All my servers have an 8 GB empty file on disk

#37
This is like carrying around a pound of beef because you refuse to look up the address of a McDonald's 7 minutes away.

Setup quotas or implement some damn monitoring -- if you're not monitoring something as simple and critical as disk usage, what else are you not monitoring?

Re: All my servers have an 8 GB empty file on disk

#38
When I worked at SevOne, we had 10x500 MB files on each disk that were called ballast files. They served the same purpose, but there were a couple nice tools built in to make sure they got repopulated when disk space was under control, plus alerting you whenever one got "blown." IIRC it could also blow ballast automatically in later versions, but I don't remember it being turned on by default.

Re: All my servers have an 8 GB empty file on disk

#40

So far, my first stop to temporarily get more disk space was to reduce the size of the swapfile which on a lot of servers seems to be allotted >1x the requirement. Will be switching to this hack! Perfect illustration of the KISS principle (Keep it simple, stupid).

Useful for people who still do 2x physical memory and you have a server with 64+ Gb
Post reply on HN