Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

All my servers have an 8 GB empty file on disk

brianschrader.com

Post reply on HN
Print view View original

51–60 of 715 posts

Re: All my servers have an 8 GB empty file on disk

#51

Post by throw0101a »

AlisdairO wrote: post #11

One other option is increasing the reserved block count ( https://ma.ttias.be/change-reserved-blocks-ext3-ext4-filesys... ). This has the nice side effect of increasing the space available for critical daemons. If you haven't customised this, in a pinch you can still lower it down a bit to buy some time.

ZFS has explicit reservations:

> The minimum amount of space guaranteed to a dataset and its descendants. When the amount of space used is below this value, the dataset is treated as if it were taking up the amount of space specified by its reservation. Reservations are accounted for in the parent datasets' space used, and count against the parent datasets' quotas and reservations.

* https://openzfs.github.io/openzfs-docs/man/8/zfsprops.8.html

These are done on a per dataset basis (basically a directory delineated boundary).

Re: All my servers have an 8 GB empty file on disk

#52

Post by uniformlyrandom »

clipradiowallet wrote: post #44

An alternative approach here... make sure (all) your filesystems are on top of LVM. This reduces the steps needed to grow your free space. Whether you have a 8gb empty file laying around, or an 8gb block device to attach...LVM will happily take them both as pv's, add them to your vg's, and finally expand your lv's. some reading if LVM is new and you want to know more: https://opensource.com/business/16/9/linux-users-…

I would not say this is an alternative, more like yet another tool in a shed:

  1. Tunefs
  2. spacer.8gb
  3. lvm

Re: All my servers have an 8 GB empty file on disk

#53

Post by ttyprintk »

clipradiowallet wrote: post #44

An alternative approach here... make sure (all) your filesystems are on top of LVM. This reduces the steps needed to grow your free space. Whether you have a 8gb empty file laying around, or an 8gb block device to attach...LVM will happily take them both as pv's, add them to your vg's, and finally expand your lv's. some reading if LVM is new and you want to know more: https://opensource.com/business/16/9/linux-users-…

Added benefit of not waiting to backup and restore 8 GB.

Re: All my servers have an 8 GB empty file on disk

#56

Post by kelnos »

gfody wrote: post #32

having an 8gb file you know you can delete isn't really all that helpful if everything has already gone disk-full-fracked. you should really have an alarm on free space, especially if you're an indie.

Sure, but sometimes the disk filling up is caused by something runaway and fast. If your "60% full" alarm goes off and the disk fills up 2 minutes later, you're still stuck.

With a "ballast file" (as another commenter termed it), you can decide exactly when processes get to start consuming disk again, and that can give you some headroom to fix the problem.

Re: All my servers have an 8 GB empty file on disk

#57

Post by 295310e0 »

SethTro wrote: post #31

Same idea as this game development legend https://www.dodgycoder.net/2012/02/coding-tricks-of-game-dev... > he had put aside those two megabytes of memory early in the development cycle. He knew from experience that it was always impossible to cut content down to memory budgets, and that many projects had come close to failing because of it. So now, as a regular practice, he always put aside a nice block of memory to…

If true, I hate that story. Think of the better art assets that were needlessly left behind. How is it that said block of memory had never been identified by any profiling?

Re: All my servers have an 8 GB empty file on disk

#58

Post by Bost »

SethTro wrote: post #31

Same idea as this game development legend https://www.dodgycoder.net/2012/02/coding-tricks-of-game-dev... > he had put aside those two megabytes of memory early in the development cycle. He knew from experience that it was always impossible to cut content down to memory budgets, and that many projects had come close to failing because of it. So now, as a regular practice, he always put aside a nice block of memory to…

There's a difference between "The server is not responding right now. We're loosing customers.", and "Low resources during product development". Actually the latter may be a case of enforcing premature optimization. So no, it's not the same idea.

Re: All my servers have an 8 GB empty file on disk

#60

Post by GekkePrutser »

chousuke wrote: post #45
GekkePrutser wrote: post #35

Earlier quoted context omitted.

I know about this, but I do think it's not a bad idea doing what he does because the reserved block count is for root and most server processes still run as root. And it's usually them that are causing the disk to fill. Though I suppose this also makes the problem itself more prominent in the first place. I guess if you run into this a lot, stricter monitoring would be a better solution. The way I found out about it…

What servers usually run as root? Some may start as root, but usually drop privileges for the actual server processes quickly, eg. apache, nginx, sshd. Nothing that actually does the "serving" or accesses data should be running as root.

No but the logfile writers are usually running as root AFAIK. And this is what tends to fill up the disk.
Post reply on HN