Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

All my servers have an 8 GB empty file on disk

brianschrader.com

Post reply on HN
Print view View original

631–640 of 715 posts

Re: All my servers have an 8 GB empty file on disk

#631

Post by bostonsre »

luckylion wrote: post #570
bostonsre wrote: post #431

Earlier quoted context omitted.

It allows me to remove that big file, then I'm able to run sudo since I don't allow root ssh and sudo won't work with a full disk, then I can clear up space on the system, bring it up again, then update log rotate or do whatever to prevent that case from happening again.

That sounds a lot more complicated (and time consuming) than just having monitoring in place, realizing the disk is filling up and fixing it before it leads to downtime.

Monitoring is in place and usually it is caught in time. Downtime is acceptable in this environment, I don't think its worth being woken up in the middle of the night when it can just be resolved in the morning.

Re: All my servers have an 8 GB empty file on disk

#632

Post by kortex »

sillysaurusx wrote: post #531
kortex wrote: post #466

Earlier quoted context omitted.

/var gets its own disk on my machine! Yay academic machine learning docker containers. What's a few 30GB docker images between friends? So yeah /var gets to live with training data on SSD which gets cold swapped onto spinny platters as needed. /home is another, on the "main sdd". /usr isn't a bad choice, either.

Yay academic machine learning docker containers. What's a few 30GB docker images between friends? :) Hopefully docker will lose the fight for mindshare in the ML space.

What do you mean, "lose mindshare"? Docker is one of the absolute best things to happen to ML. My work is largely algo integration - I take academics' ML code and bundle it for API consumption. Before this, I was a bench chemist, so I kind of have a thing for reproducing experiments.

The academic ML scene has a reproducibility crisis that makes other science reproducibility crises look like Phys 100 labs. These things depend on someone's conda env with versions pinned to nightly builds, inscrutable code written by a postdoc that has since left, datasets which have been mutated since the paper was published, on and on.

Docker gives me a fighting chance to actually get reproducible results without going stark raving mad.

Should these researchers have better software process to avoid this situation in the first place? Heck yes. But these are people doing things like `os.system("rm " + filepath)` and git committing entire models, they are just really green and most don't know any better yet and academia isn't really known for its mandatory Practical Software Engineering classes for CS majors.

Re: All my servers have an 8 GB empty file on disk

#633

Post by scottlamb »

herpderperator wrote: post #598
ryandrake wrote: post #147

A lot of tips in this thread are about how to better alert when you get low on disk space, how to recover, etc. but I'd like to highlight the statement: "The disk filled up, and that's one thing you don't want on a Linux server—or a Mac for that matter. When the disk is full nothing good happens." As developers, we need to be better at handling edge cases like out of disk space, out of memory, pegged bandwidth and pe…

> As developers, we need to be better at handling edge cases like out of disk space, out of memory, pegged bandwidth and pegged CPU In what situation though? Let's consider disk space. This certainly does not apply to all developers or all programs. Making your program understand the fact that the system has no space left does not seem like something that would be very productive in the vast majority of cases. Like r…

It's important that you can recover from this condition.

For example, I'm working on an NVR project. It has a SQLite database that should be placed on your SSD-based root filesystem and puts video frames on spinning disks. It's essentially a specialized DBMS. You should never touch its data except though its interface.

If you misconfigure it, it will fill the spinning disks and stall. No surprise there. The logical thing for the admin to do is stop it, go into the config tool, reduce the retention, and restart. (Eventually I'd like to be able to reconfigure a running system but for now this is fine.)

But...in an earlier version, this wouldn't work. It updates a small metadata file in each video dir on startup to help catch accidents like starting with an older version of the db than the dir or vice versa. It used to do this by writing a new metadata file and then renaming into place. This procedure would fail and you couldn't delete anything. Ugh.

I fixed it through a(nother) variation of preallocation. Now the metadata files are a fixed 512 bytes. I just overwrite them directly, assuming the filesystem/block/hardware layers offer atomic writes this size. I'm not sure this assumption is entirely true (you really can't find an authoritative list of filesystem guarantees, unfortunately), but it's more true then assuming disks never fill.

It might also not start if your root filesystem is full because it expects to be able to run SQLite transactions, which might grow the database or WAL. I'm not as concerned about this. The SQLite db is normally relatively small and you should have other options for freeing space on the root filesystem. Certainly you could keep a delete-me file around as the author does.

Re: All my servers have an 8 GB empty file on disk

#634

Post by splithalf »

rgj wrote: post #79

In the mid nineties I worked in a research institute. There was a large shared Novell drive which was always on the verge of full. Almost every day we were asked to clean up our files as much as possible. There were no disc quota for some reason. One day I was working with my colleague and when the fileserver was full he went to a project folder and removed a file called balloon.txt which immediately freed up a few p…

This still occurs.

Re: All my servers have an 8 GB empty file on disk

#635

Post by hprotagonist »

wingerlang wrote: post #509
hprotagonist wrote: post #302

Earlier quoted context omitted.

The bikes I've had that have had reserve tanks have also been old enough to raise the disconcerting follow-on question, which is: "is the reserve gas also full of sludgey crap that's settled in the tank and hasn't been disturbed really in a year, and am i about to run that through my poor carbs?"

It’s not a separate tank (in any of my bikes at least) so it gets disturbed every time you refill the tank?

the two-tube design of the tank on my 1975 honda CB meant that there was about an inch and a half of tank that sat below the primary fuel port. Tank crud (steel tank, theoretically passivated, 40 years old) settles faster than I ran through a tank of gas, so the bottom layer had sediment in it fairly regularly.

I kept spare inline fuel filters in a tool roll just in case after a while.

Re: All my servers have an 8 GB empty file on disk

#636

Post by bombcar »

tyingq wrote: post #285
zepearl wrote: post #258

Earlier quoted context omitted.

I think that in the past I saw that when creating a file with e.g. ... dd if=/dev/zero of=deleteme.file bs=1M count=8196 ...the "free space" shown by "df" slowly decreased while the file was being created, but then once the operation completed that "free space" magically went back to its original value => the big existing file (full of "0"s) was basically not using any storage. Is this what you mean? I just tried to…

dd will create sparse files if you use the seek option, like: dd if=/dev/zero of=a_sparse_file bs=1 count=0 seek=8G

You have to beware if you're on a filesystem (such as ZFS) that has compression enabled. A file of all zeros compresses quite well, and may not get you the space you need when you remove it.

Re: All my servers have an 8 GB empty file on disk

#637

Post by kmonsen »

HelloNurse wrote: post #562
kmonsen wrote: post #487

Earlier quoted context omitted.

I was in the army, they had a certain budget for bullets. Last day of the year we shoot the rest because if not they would get less next year. We shot so much we destroyed some of the rifles, apparently that was better than getting a smaller allocation next year.

Did you have a budget for replacement rifles too? Unlike bullets, there might be some incentive to get new and different ones.

That I don’t know, they were all replaced with the same (old) version.

Re: All my servers have an 8 GB empty file on disk

#638

Post by downrightmike »

njovin wrote: post #335
victor9000 wrote: post #163

Earlier quoted context omitted.

Perfect example of the tragedy of the commons. If individuals don't create these balloon files then they won't be able to use the file server when they need it, yet by creating these balloon files the collective action depletes the shared resource of its main function.

This is similar to how some government agencies retain their budgets. At the end of the budget period they've only spent 80% of their allocated budget, so they throw out a bunch of perfectly good equipment/furniture/etc. and order new stuff so that their budget doesn't get cut the following year, rather than accepting that maybe they were over-budgeted to begin with. Rinse, repeat, thus continuing the cycle of wastin…

That's why the US Navy dump perfectly fine fuel overboard.

Re: All my servers have an 8 GB empty file on disk

#639

Post by zepearl »

tyingq wrote: post #285
zepearl wrote: post #258

Earlier quoted context omitted.

I think that in the past I saw that when creating a file with e.g. ... dd if=/dev/zero of=deleteme.file bs=1M count=8196 ...the "free space" shown by "df" slowly decreased while the file was being created, but then once the operation completed that "free space" magically went back to its original value => the big existing file (full of "0"s) was basically not using any storage. Is this what you mean? I just tried to…

dd will create sparse files if you use the seek option, like: dd if=/dev/zero of=a_sparse_file bs=1 count=0 seek=8G

Thank you. I actually never used that seek-option, but I still think that what I wrote did happen. Mistery... .

Re: All my servers have an 8 GB empty file on disk

#640

Post by zepearl »

bombcar wrote: post #636
tyingq wrote: post #285

Earlier quoted context omitted.

dd will create sparse files if you use the seek option, like: dd if=/dev/zero of=a_sparse_file bs=1 count=0 seek=8G

You have to beware if you're on a filesystem (such as ZFS) that has compression enabled. A file of all zeros compresses quite well, and may not get you the space you need when you remove it.

Yep - but I think that I was using ext4 and/or xfs when I was seeing that happening... .
Post reply on HN