Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

All my servers have an 8 GB empty file on disk

brianschrader.com

Post reply on HN
Print view View original

531–540 of 715 posts

Re: All my servers have an 8 GB empty file on disk

#531

Post by sillysaurusx »

kortex wrote: post #466
yabones wrote: post #112

Earlier quoted context omitted.

100% agree. I think at the bare minimum every system should have two partitions: `/` and `/var` /var is usually where the most data gets added. Logs, database files, caches, and whatever other junk your app spits out. 99% of the time that's what causes the out of space issues. By keeping that separate from root, you've saved your system from being completely hosed when it fills up (which it will). Obviously there are…

/var gets its own disk on my machine! Yay academic machine learning docker containers. What's a few 30GB docker images between friends? So yeah /var gets to live with training data on SSD which gets cold swapped onto spinny platters as needed. /home is another, on the "main sdd". /usr isn't a bad choice, either.

Yay academic machine learning docker containers. What's a few 30GB docker images between friends?

:)

Hopefully docker will lose the fight for mindshare in the ML space.

Re: All my servers have an 8 GB empty file on disk

#532

Post by AYBABTME »

Once upon a time, I wanted to cache large and expensive to pull files on many thousands of servers. Problem is the disk space on these servers was at premium and meant to be sold for customer use. The servers did have scratch space on small disks, but that was used by the OS.

So I wrote an on-disk cache system that would monitor disk usage, and start to evict and shrink its disk space usage. It would take up to N gigabytes of disk (configurable) for the purpose of caching, and maintain an M gigabytes free-disk-space buffer.

Say you had a 100 GiB total space on a partition, with 8 GiB used for cache with a 2 GiB headroom. As legitimate/regular (customer) space usage increased and reached 91 GiB, the cache would see 9 GiB available, and removing the 2 GiB buffer, would start to evict items to resize to 7 GiB, and so on until it had evicted everything.

When this system deployed, it started to trigger low-disk-space alerts earlier than before. At first that seemed like a problem, but the outcome is that we were now getting low-disk-space alerts with more advance warning, and the cache bought some time as it kept resizing down to free up space. It kind of, in a way, served the same purpose as described in this blog post.

Overall this cache was pretty neat and still is, I bet. There's probably ways to do similar things with fancy filesystems (or obscure features) but this was a quick thing to deploy across all servers without having to change any system setting or change the filesystem.

I sometime wish I had done this in open-source, because it would be convenient to use locally on my laptop, or on many servers.

Re: All my servers have an 8 GB empty file on disk

#533

Post by larusso »

I maintain a small fleet of CI machines (mostly macs) and run into this issue as well from time to time. The free space idea is nice but I ram into the problem that under very critical disk space I can’t even shh or when delete a file because there is simply not enough space to execute the simple command. A reboot to get rid of some temp files helps me in these situations to get some control back.

Re: All my servers have an 8 GB empty file on disk

#534

Post by bottled_poe »

Aeolun wrote: post #383
njovin wrote: post #335

Earlier quoted context omitted.

This is similar to how some government agencies retain their budgets. At the end of the budget period they've only spent 80% of their allocated budget, so they throw out a bunch of perfectly good equipment/furniture/etc. and order new stuff so that their budget doesn't get cut the following year, rather than accepting that maybe they were over-budgeted to begin with. Rinse, repeat, thus continuing the cycle of wastin…

I think the problem is that you do not need 100% of your budget every year, but getting it back when you do need it is much harder than keeping it in the first place. Definite case of misaligned incentives.

Why aren’t there rolling budgets over multiple years with annual reviews on spend?

Re: All my servers have an 8 GB empty file on disk

#535

Post by t0mmel »

What I don’t understand about this approach is why you think it actually does anything for you ? What you do instead of this is to setup an alert to monitor disk space at the right threshold for you, and then have a contingency plan for how to add more space to your environment.

It seems like you have sort of done that, but in this case you are actually allowing your system to get into a bad state before you react.

Perhaps it’s better to be proactive instead of reactive.

Re: All my servers have an 8 GB empty file on disk

#536

Post by nottorp »

herpderperator wrote: post #473

The author seems to forget that ext-based filesystems keep 5% of disk space available for root at all times by default, known as "reserved blocks".[0] That means if a non-root user uses all of the available space, it wasn't really all of the space -- root still has access to 5% free space within that partition. That's exactly the same as the useless 8GB file but in an officially-supported manner. If you run out of di…

> So log in as root and fix the issue. Simple.

Except you can't ssh in as root any more. You have to ssh as a normal user then sudo in. Which may not be possible.

A paid on call admin may not mind driving to the data center too much. Everyone else would.

Re: All my servers have an 8 GB empty file on disk

#537

Post by ganafagol »

ben509 wrote: post #208
luckylion wrote: post #131

Earlier quoted context omitted.

> 1) It's not intended to be a Real Solution(tm). It's intended to buy the admin some time to solve the Real Issue. If you don't have monitoring, will you even be aware that your disk is filling up? If you do have monitoring, why are you artificially filling up your disk so that it will be at 100% more quickly instead of just setting your monitoring up to alert you when it's at $whateverItWasSetToMinusEightGB?

One argument in favor of it is the 8GB file may cause a runaway process to crash, leaving you without it continuing to chew up space and able to recover. A second argument is it's not opened by any process. One problem I've had fixing disk full errors was figuring out which process still had a file open. (For any POSIX noobs: the space occupied by a file is controlled by its inode. Deleting a file "unlinks" the inode…

lsof

Re: All my servers have an 8 GB empty file on disk

#538

Post by ridethebike »

sgregnt wrote: post #461
lostlogin wrote: post #454

Earlier quoted context omitted.

> This is similar to how some government agencies retain their budgets. The non-government sector isn’t immune to this.

If whatever actions turn a non government entity into something inefficient, then the entity wont survive for long and will go out of business (or at least that's the hope of a competitive free market economy)

It's not always the case.

I work at one of those US bank.

The amount of inefficiencies in form of red tape, confusing processes, custom half-baked tools that crash half the time is just mind-boggling. I've spent more than a week now for opening firewall on one ip/port on one host just to test my prototype in dev environment(local machine or docker are not an option due to lack of admin rights ), and it's still in change approval stage. If we weren't this giant too-big-to-fail bank we'd be out of business by now.

Re: All my servers have an 8 GB empty file on disk

#539

Post by gitgud »

t0mmel wrote: post #535

What I don’t understand about this approach is why you think it actually does anything for you ? What you do instead of this is to setup an alert to monitor disk space at the right threshold for you, and then have a contingency plan for how to add more space to your environment. It seems like you have sort of done that, but in this case you are actually allowing your system to get into a bad state before you react. P…

This is true, but alerts and monitoring software can fail in many ways, due to network issues, process crashes... Etc.

Due to the simplicity of this solution, there's not much that can go wrong.... unless you can't actually access the server anymore to delete the file...

I see this more as a second line of defence

Re: All my servers have an 8 GB empty file on disk

#540

Post by noisy_boy »

skissane wrote: post #382
njovin wrote: post #335

Earlier quoted context omitted.

This is similar to how some government agencies retain their budgets. At the end of the budget period they've only spent 80% of their allocated budget, so they throw out a bunch of perfectly good equipment/furniture/etc. and order new stuff so that their budget doesn't get cut the following year, rather than accepting that maybe they were over-budgeted to begin with. Rinse, repeat, thus continuing the cycle of wastin…

Years ago I worked as a research assistant for a university. One day, my boss (a professor) pulled me aside for an impromptu meeting. "I have $5000 left in a research grant I need to spend this week or else it's gone forever – do you have any ideas of what I should spend it on?" Unfortunately I couldn't think of much. I suggested maybe we buy some more computers with it but I'm sure he'd already thought of that himse…

Buy annual licenses. Then renew if you want to block the same amount next year OR don't renew and use it for what you actually need.
Post reply on HN