Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

All my servers have an 8 GB empty file on disk

brianschrader.com

Post reply on HN
Print view View original

171–180 of 715 posts

Re: All my servers have an 8 GB empty file on disk

#171

Post by [deleted] »

dilyevsky wrote: post #153
geocrasher wrote: post #124

For everyone saying "This isn't a real solution!" I'd like to explain why I think you're wrong. 1) It's not intended to be a Real Solution(tm). It's intended to buy the admin some time to solve the Real Issue. 2) Having a failsafe on standby such as this will save an admin's butt when it's 2am and PagerDuty won't shut up, and you're just awake enough to apply a temp fix and work on it in the morning. 3) Because "FIX…

Setup proper monitoring and never get to the Real Issue to begin with. These sysadmin hacks are not helpful

[deleted]

Re: All my servers have an 8 GB empty file on disk

#172

Post by viraptor »

emmab wrote: post #77
gherkinnn wrote: post #63

An architect once told me that he always plans for a solid gold block hidden away in the cellar. Once the project invariably goes over budget, he drops the plans for the gold and frees up extra funds. Edit: I think it was a large marble slab. Same thing.

What?!? How does that work? Does he just draw up a blueprint and write "solid gold block goes here" and them some contractor says "yes that gold block will be $NNNNN" and includes it in the budget??

It's likely not literal. He likely quotes for price +50k or something like that, so that people will start thinking about reducing the price before they run out of budget.

Re: All my servers have an 8 GB empty file on disk

#173

Post by Dylan16807 »

manifoldgeo wrote: post #109
gowld wrote: post #101

Earlier quoted context omitted.

You had a community-driven quota system.

Sounds like they had a tragedy of the commons, too haha

Partly that, partly the opposite.

It's basically reserving part of the disk for very important things only, which scares off less important uses. Like making the commons seem more polluted than it actually is to get some action taken.

If those files weren't there, the space would probably fill up, but now without any emergency relief valves.

It would be better if these files were a smaller fraction of space and had more oversight... but that's just a quota system. This is something halfway in between real quotas and full-on tragedy of the commons.

Re: All my servers have an 8 GB empty file on disk

#174

Post by lostcolony »

diego wrote: post #54

This points to a much more serious problem. This is 2021 and the technology is from the 90s, with a really poor user experience design. Your car warns you when you're low on fuel, but your server doesn't if you're low on critical resources.

You car also doesn't drop from "alarm" to "empty" in thirty seconds. A HD on a VM with a bad behaving process can.

Re: All my servers have an 8 GB empty file on disk

#175

Post by tytso »

birdyrooster wrote: post #47
stonesweep wrote: post #24

Earlier quoted context omitted.

I suspect the blog author did not understand this (based on the content) - as a Linode user myself, I just had a look at one of my VMs and they install with the regular 5% reserved space (ext4/Debian).

Funny because I have always tune2fs -m1 or tune2fs -m0 because the reserved space was never supposed to scale linearly with hard drive capacities and is not useful to userspace in anyway. Have never had any issues and been doing it for decades in commercial applications. In some cases, where you probably shouldn't be using ext3/4 anyways, we are talking about reclaiming TBs of reserved space. It's important to note t…

If you try to use a file system to 99% full --- and it doesn't matter whether it is a 10GB file system or a 10TB file system, you will see significant performance penalties as the file system gets badly fragmented. So that's why having a fixed percentage even for massively big disks still makes sense.

Disk space is cheap enough that even 5% of a 14TB disk is really not that much money --- and if you see bad performance, and then have to pay $$$ to switch to an SSD, maybe it would have been much cheaper to use a HDD with a larger free space reserve....

Re: All my servers have an 8 GB empty file on disk

#176

Post by jodrellblank »

cure wrote: post #129
qw3rty01 wrote: post #88

Earlier quoted context omitted.

One VM using excessively more disk space than it's supposed to can potentially cause data corruption in all the other VMs on that system. For just spinning VMs up and down for testing, you probably won't run into that issue, but on a production system, it could potentially cause some massive downtime

Virtual machine disk space (e.g. Xen, Linode, AWS EC2, or similar) does not work this way. Each VM gets a dedicated amount of disk space allocated to it, they don't all share a pool of free space.

Yes they do with the "dyanmic allocation" the parent comment mentions; VMware datastore has 1TB total, you put VMs in with dynamically expanding disks they are sharing the same 1TB of free space and will fill it if they all want their max space at the same time and you've overprovisioned their max space.

And if you haven't overprovisioned their max space, you may as well not be using dynamic allocation and use fixed size disks.

Even then, snapshots will grow forever and fill the space, and then you hope you have a "spacer.img" file you can delete from the datastore, because you can't remove snapshots when the disk is full and you're stuck. It's the same problem, at a lower level.

Re: All my servers have an 8 GB empty file on disk

#177

Post by qwertox »

jbverschoor wrote: post #68
mgarfias wrote: post #55

Why not keep an eye on the disk and expand the fs before it goes south?

Because sometimes it can fill up quite quickly. And the extra space will give to the headroom you need. Will def do this on all my servers

Yes. I like the idea very much and also think that I will do this with a couple of my machines.

Re: All my servers have an 8 GB empty file on disk

#179

Post by [deleted] »

raldi wrote: post #117

This reminds me of Perl’s esoteric $^M variable. You assign it some giant string, and in an out-of-memory condition, the value is cleared to free up some emergency space for graceful shutdown. “To discourage casual use of this advanced feature, there is no English long name for this variable.” But the language-build flag to enable it has a great name: -DPERL_EMERGENCY_SBRK, obviously inspired by emergency brake.

[deleted]

Re: All my servers have an 8 GB empty file on disk

#180

Post by labawi »

jonhermansen wrote: post #116
clipradiowallet wrote: post #44

An alternative approach here... make sure (all) your filesystems are on top of LVM. This reduces the steps needed to grow your free space. Whether you have a 8gb empty file laying around, or an 8gb block device to attach...LVM will happily take them both as pv's, add them to your vg's, and finally expand your lv's. some reading if LVM is new and you want to know more: https://opensource.com/business/16/9/linux-users-…

If you are using LVM on all of your filesystems, it seems like a bad idea to use a file residing on LVM block device as another PV. And actually I'd be surprised if this was even allowed. Though maybe it is difficult to detect. You'd effectively send all block changes through LVM twice (once through the file, then through the underlying block device(s))

LVM is just fancy orchestration for the device-mapper subsystem with some headers for setup information.

For block operations it's no different from manual setup of loop-mounted volumes, that also need to travel a couple of layers to hit the backing device.

Though there is an important caveat - LVM is more abstracted, making it easier to mistakenly map a drive onto itself, which may create a spectacular failure (haven't tried).

Post reply on HN