Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

All my servers have an 8 GB empty file on disk

brianschrader.com

Post reply on HN
Print view View original

141–150 of 715 posts

Re: All my servers have an 8 GB empty file on disk

#141

Post by tehjoker »

geocrasher wrote: post #124

For everyone saying "This isn't a real solution!" I'd like to explain why I think you're wrong. 1) It's not intended to be a Real Solution(tm). It's intended to buy the admin some time to solve the Real Issue. 2) Having a failsafe on standby such as this will save an admin's butt when it's 2am and PagerDuty won't shut up, and you're just awake enough to apply a temp fix and work on it in the morning. 3) Because "FIX…

This reminds me of the reserve tank toggle on some motorcycles. When you run out of gas, you switch the toggle and drive directly to a gas station.

Re: All my servers have an 8 GB empty file on disk

#143

Post by stonesweep »

birdyrooster wrote: post #47
stonesweep wrote: post #24

Earlier quoted context omitted.

I suspect the blog author did not understand this (based on the content) - as a Linode user myself, I just had a look at one of my VMs and they install with the regular 5% reserved space (ext4/Debian).

Funny because I have always tune2fs -m1 or tune2fs -m0 because the reserved space was never supposed to scale linearly with hard drive capacities and is not useful to userspace in anyway. Have never had any issues and been doing it for decades in commercial applications. In some cases, where you probably shouldn't be using ext3/4 anyways, we are talking about reclaiming TBs of reserved space. It's important to note t…

Fun trivia: you can actually set per-device settings in /etc/mke2fs.conf with all sorts of alternate defaults. https://man7.org/linux/man-pages/man5/mke2fs.conf.5.html

Re: All my servers have an 8 GB empty file on disk

#144

Post by znpy »

That's a dumb idea?

Iirc some filesystems allow you to reserve a percentage of blocks for this particular use case (recovery by root).

Ext2/3 for sure, ext4 probably too.

Not sure you can do that on linode on the rootfs, since the filesystem is mounted, tho.

Re: All my servers have an 8 GB empty file on disk

#145

Post by [deleted] »

luckylion wrote: post #131
geocrasher wrote: post #124

For everyone saying "This isn't a real solution!" I'd like to explain why I think you're wrong. 1) It's not intended to be a Real Solution(tm). It's intended to buy the admin some time to solve the Real Issue. 2) Having a failsafe on standby such as this will save an admin's butt when it's 2am and PagerDuty won't shut up, and you're just awake enough to apply a temp fix and work on it in the morning. 3) Because "FIX…

> 1) It's not intended to be a Real Solution(tm). It's intended to buy the admin some time to solve the Real Issue. If you don't have monitoring, will you even be aware that your disk is filling up? If you do have monitoring, why are you artificially filling up your disk so that it will be at 100% more quickly instead of just setting your monitoring up to alert you when it's at $whateverItWasSetToMinusEightGB?

[deleted]

Re: All my servers have an 8 GB empty file on disk

#147

Post by ryandrake »

A lot of tips in this thread are about how to better alert when you get low on disk space, how to recover, etc. but I'd like to highlight the statement: "The disk filled up, and that's one thing you don't want on a Linux server—or a Mac for that matter. When the disk is full nothing good happens."

As developers, we need to be better at handling edge cases like out of disk space, out of memory, pegged bandwidth and pegged CPU. We typically see the bug in our triage queue and think in our minds "Oh! out of disk space: Edge case. P3. Punt it to the backlog forever." This is how we get in this place where every tool in the toolbox simply stops working when there's zero disk space.

Especially on today's mobile devices, running out of disk space is common. I know people who install apps, use them, then uninstall them when they're done, in order to save space, because their filesystem is choked with thousands of pictures and videos. It's not an edge case anymore, and should not be treated as such.

Re: All my servers have an 8 GB empty file on disk

#148

Post by londons_explore »

mamon wrote: post #135
londons_explore wrote: post #120

Linux has this built in... By default, only root can use the last 5% of disk space. That means you can fire up a root shell and know you have a buffer of free space to resolve the issue.

But now we have Docker, which means all the containerized workflows will run as root....

I suspect you need to be root in the root user namespace... So docker doesn't get this special power...

Re: All my servers have an 8 GB empty file on disk

#149

Post by tomrod »

rgj wrote: post #79

In the mid nineties I worked in a research institute. There was a large shared Novell drive which was always on the verge of full. Almost every day we were asked to clean up our files as much as possible. There were no disc quota for some reason. One day I was working with my colleague and when the fileserver was full he went to a project folder and removed a file called balloon.txt which immediately freed up a few p…

This is very common in F500 companies. It's also a symptom of dysfunction.

Re: All my servers have an 8 GB empty file on disk

#150

Post by da_big_ghey »

Full disc problem in linux macine has been a problem in partialy solved in past many decades. We have had seperated partition /home, /tmp, /var, /usr in each its own partition. This is reduce problem if not completly removing. This is small desadvantage: there is reducion in fungability for a disc space.
Post reply on HN