Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

All my servers have an 8 GB empty file on disk

brianschrader.com

Post reply on HN
Print view View original

131–140 of 715 posts

Re: All my servers have an 8 GB empty file on disk

#131

Post by luckylion »

geocrasher wrote: post #124

For everyone saying "This isn't a real solution!" I'd like to explain why I think you're wrong. 1) It's not intended to be a Real Solution(tm). It's intended to buy the admin some time to solve the Real Issue. 2) Having a failsafe on standby such as this will save an admin's butt when it's 2am and PagerDuty won't shut up, and you're just awake enough to apply a temp fix and work on it in the morning. 3) Because "FIX…

> 1) It's not intended to be a Real Solution(tm). It's intended to buy the admin some time to solve the Real Issue.

If you don't have monitoring, will you even be aware that your disk is filling up?

If you do have monitoring, why are you artificially filling up your disk so that it will be at 100% more quickly instead of just setting your monitoring up to alert you when it's at $whateverItWasSetToMinusEightGB?

Re: All my servers have an 8 GB empty file on disk

#132

Post by amock »

raldi wrote: post #117

This reminds me of Perl’s esoteric $^M variable. You assign it some giant string, and in an out-of-memory condition, the value is cleared to free up some emergency space for graceful shutdown. “To discourage casual use of this advanced feature, there is no English long name for this variable.” But the language-build flag to enable it has a great name: -DPERL_EMERGENCY_SBRK, obviously inspired by emergency brake.

I think it likely relates to https://en.m.wikipedia.org/wiki/Sbrk.

Re: All my servers have an 8 GB empty file on disk

#133

Post by copperfoil »

rozap wrote: post #118
diego wrote: post #54

This points to a much more serious problem. This is 2021 and the technology is from the 90s, with a really poor user experience design. Your car warns you when you're low on fuel, but your server doesn't if you're low on critical resources.

Everyone has this kind of alerting set up, but that's not the point. The beauty of this solution is that it's dead simple and will never fail. Alerting can fail or be ignored. It's the same as old VW beetles which had a reserve gas tank. When you ran out of gas you opened a valve and you could limp to a gas station. Less likely to fail versus a 1950's era gauge that is telling you you're low. Also impossible to ignor…

> The beauty of this solution is that it's dead simple and will never fail. Alerting can fail or be ignored.

It's not that straightforward IMO. Would this file be deleted before the space is filled? If so, there is alerting in place, and it assumes there's a way to delete files before space fills up. If this file is deleted after space fills up, how is this different from not having the file, other than making finding files to delete easier? Then what happens after that? If you delete the file and realize there's nothing else to delete, you'd have to solve the problem the same way if you didn't use this method.

Re: All my servers have an 8 GB empty file on disk

#135

Post by mamon »

londons_explore wrote: post #120

Linux has this built in... By default, only root can use the last 5% of disk space. That means you can fire up a root shell and know you have a buffer of free space to resolve the issue.

But now we have Docker, which means all the containerized workflows will run as root....

Re: All my servers have an 8 GB empty file on disk

#136

Post by ineedasername »

A great idea, but it still leaves the possibility for performance issues prior to an admin's ability to address is. Some like two 4gb blocks might work better: if you get within, say, 200mb of storage limits you remove the first one and trigger an email/text/whatever to the admin, that way they can address it before it goes further. It's an early warning and automated solution. Then, if the situation continues, the second 4gb block is also automatically removed with another message send to the admin. Nothing fails silently.

Re: All my servers have an 8 GB empty file on disk

#137

Post by nijave »

yabones wrote: post #112
danielrhodes wrote: post #80

My understanding is this is why one should partition a drive. If you have a data partition, a swap partition, and an OS partition, you can get around issues where a server’s lack of disk space hoses the whole system.

100% agree. I think at the bare minimum every system should have two partitions: `/` and `/var` /var is usually where the most data gets added. Logs, database files, caches, and whatever other junk your app spits out. 99% of the time that's what causes the out of space issues. By keeping that separate from root, you've saved your system from being completely hosed when it fills up (which it will). Obviously there are…

I place I used to work achieved something similar with lvm thin provisioning and split out something like /, /home, /var, /var/log and maybe a couple others. I think they also had something clever with lvm snapshots to rollback bad updates (snapshot system, upgrade, verify) so even if an update went rogue and deleted some important, unrelated files it could be undid

Re: All my servers have an 8 GB empty file on disk

#139

Post by raldi »

amock wrote: post #132
raldi wrote: post #117

This reminds me of Perl’s esoteric $^M variable. You assign it some giant string, and in an out-of-memory condition, the value is cleared to free up some emergency space for graceful shutdown. “To discourage casual use of this advanced feature, there is no English long name for this variable.” But the language-build flag to enable it has a great name: -DPERL_EMERGENCY_SBRK, obviously inspired by emergency brake.

I think it likely relates to https://en.m.wikipedia.org/wiki/Sbrk .

Yes, of course. That’s what makes the pun work.
Post reply on HN