Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

All my servers have an 8 GB empty file on disk

brianschrader.com

Post reply on HN
Print view View original

331–340 of 715 posts

Re: All my servers have an 8 GB empty file on disk

#331

Post by joshjdr »

icansearch wrote: post #312
plank_time wrote: post #304

Earlier quoted context omitted.

I bought a bunch of rice, and all that happened was little rice bugs started living in it, so I had to throw it all away. But if there were a rice shortage, I would probably have eaten it.

If you freeze it for 24 hours it will kill them off. Can be a good idea to do that when it comes into the house anyway as they might be in there already. Easy enough to separate them out of the rice after freezing too.

Extra protein (not to mention a few deviations from the OP)?

Re: All my servers have an 8 GB empty file on disk

#332

Post by ineedasername »

tshaddox wrote: post #327
ineedasername wrote: post #155

Earlier quoted context omitted.

Because even if you have monitoring, some unforseen issue rapidly eating disk space at 3:00 am may not give you the time to solve it without downtime or degraded performance unless you can immediately remove the bottleneck while you troubleshoot.

Then why not automate the removal of the 8 GB spacer file when the disk gets full? Or in other words, just sound your alarms when there is 8 GB of free disk space.

I actually suggested exactly that in another comment, thoigtnto do it in stages: 4gb with an alarm, the more alarms and the other 4gb if not resolved.

Re: All my servers have an 8 GB empty file on disk

#333

Post by pjmorris »

SethTro wrote: post #31

Same idea as this game development legend https://www.dodgycoder.net/2012/02/coding-tricks-of-game-dev... > he had put aside those two megabytes of memory early in the development cycle. He knew from experience that it was always impossible to cut content down to memory budgets, and that many projects had come close to failing because of it. So now, as a regular practice, he always put aside a nice block of memory to…

I'd read in 'Apollo: Race To The Moon', Murray and Cox, that the booster engineers had done something similar with their weight budget, something the spacecraft engineers wound up needing. Contingency funds of all sorts are a great thing.

Re: All my servers have an 8 GB empty file on disk

#335

Post by njovin »

victor9000 wrote: post #163
rgj wrote: post #79

In the mid nineties I worked in a research institute. There was a large shared Novell drive which was always on the verge of full. Almost every day we were asked to clean up our files as much as possible. There were no disc quota for some reason. One day I was working with my colleague and when the fileserver was full he went to a project folder and removed a file called balloon.txt which immediately freed up a few p…

Perfect example of the tragedy of the commons. If individuals don't create these balloon files then they won't be able to use the file server when they need it, yet by creating these balloon files the collective action depletes the shared resource of its main function.

This is similar to how some government agencies retain their budgets.

At the end of the budget period they've only spent 80% of their allocated budget, so they throw out a bunch of perfectly good equipment/furniture/etc. and order new stuff so that their budget doesn't get cut the following year, rather than accepting that maybe they were over-budgeted to begin with.

Rinse, repeat, thus continuing the cycle of wasting X% of the budget every year.

Re: All my servers have an 8 GB empty file on disk

#336

Post by easton »

ABeeSea wrote: post #300
twistedpair wrote: post #261

Earlier quoted context omitted.

I worked at a large company during a migration from Lotus to Outlook. We were told we'd get our current Lost email storage + 100MiB as a new email quota limit under Outlook. I made a bunch of 100MiB files of `/dev/random` noise (so they don't compress, compressed size was part of the quota) and emailed them to myself before the migration, to get a few GiB of quota buffer. My co-workers were constantly having to delet…

Email quotas aren’t just a cost thing. It forces deletion of files/communications that aren’t relevant anymore. The last thing the legal department wants is some executive’s laptop with 10 years of undeleted email to make it’s way to discovery.

Then why not just tell Exchange to delete any emails older than 5 years (or whatever your lawyers tell you to put)?

Re: All my servers have an 8 GB empty file on disk

#338

Post by ChuckMcM »

rgj wrote: post #79

In the mid nineties I worked in a research institute. There was a large shared Novell drive which was always on the verge of full. Almost every day we were asked to clean up our files as much as possible. There were no disc quota for some reason. One day I was working with my colleague and when the fileserver was full he went to a project folder and removed a file called balloon.txt which immediately freed up a few p…

Okay, that is hilarious.

I use some scripts that monitor disk space, and monitor disk usages by "subsystem" (logs, mail, services, etc) using Nagios. And as DevOps Borat says, "Disk not full unless Nagios say 'Disk is full'" :_) Although long before it is full it starts warning me.

It doesn't go off very much, but it did when I had a bunch of attacks on my web server that started core dumping and that filled up disk reasonably quickly.

Back in the day we actually put different things in different partitions so that we could partition failures but that seems out of favor with a lot of the distros these days.

Re: All my servers have an 8 GB empty file on disk

#339

Post by jeroenhd »

bostonsre wrote: post #78
scottlamb wrote: post #65

> On Linux servers it can be incredibly difficult for any process to succeed if the disk is full. Copy commands and even deletions can fail or take forever as memory tries to swap to a full disk and there's very little you can do to free up large chunks of space. This reasoning doesn't make sense. On Linux, swap is preallocated. This is true regardless of whether you're using a swap partition or a swap file. See man…

Not sure about their reasoning.. but if you don't have root ssh enabled, sudo can break if there is no free disk space. I do something similar where I write a 500mb file to /tmp and chmod 777 it so anyone can free it up without needing sudo.

I've never had sudo break on my full disks. However, that doesn't mean recovery is easy...

Working in a terminal to find out what on earth has just filled up your disk is a real pain when your shell complains about failing to write to your $HISTFILE and such. And, of course, the problem always shows up on that one server that doesn't have ncdu installed...

I'm sure sudo can theoretically break with 0 free disk space, but that's not the usual mode of failure in my experience. At most sudo need to touch a dotfile or two, so deleting _any_ temporary file or old log archive will do for it to recover.

The balloon file is not a bad idea. I think I will apply it on my own servers just for good measure, although 8GiB is a bit much for my tastes.

Re: All my servers have an 8 GB empty file on disk

#340

Post by apocalyptic0n3 »

luckylion wrote: post #131
geocrasher wrote: post #124

For everyone saying "This isn't a real solution!" I'd like to explain why I think you're wrong. 1) It's not intended to be a Real Solution(tm). It's intended to buy the admin some time to solve the Real Issue. 2) Having a failsafe on standby such as this will save an admin's butt when it's 2am and PagerDuty won't shut up, and you're just awake enough to apply a temp fix and work on it in the morning. 3) Because "FIX…

> 1) It's not intended to be a Real Solution(tm). It's intended to buy the admin some time to solve the Real Issue. If you don't have monitoring, will you even be aware that your disk is filling up? If you do have monitoring, why are you artificially filling up your disk so that it will be at 100% more quickly instead of just setting your monitoring up to alert you when it's at $whateverItWasSetToMinusEightGB?

Besides runaway log files that aren't being properly rotated, human error can cause it too. I managed to completely eat up the disk space of one of our staging servers a few weeks ago trying to tar up a directory so I could work on it locally. Didn't realize the directory was 31GB and we only had 25GB of space. By the time the notification for 80% usage was triggered (no more than 2 minutes after we hit 80%), the entire disk was full. Luckily it was just a staging server and no real harm was done, but such a mistake could have just as easily been made on a production server. In this case, the obvious solution is to just delete the file you were creating but if you're running a more complicated process that is generating logs and many files, it may not be so easy and this 8GB empty file might be useful after you cancel the process.
Post reply on HN