Live data from Hacker News

All my servers have an 8 GB empty file on disk

brianschrader.com

241–250 of 715 posts

Re: All my servers have an 8 GB empty file on disk

#241

For everyone saying "This isn't a real solution!" I'd like to explain why I think you're wrong. 1) It's not intended to be a Real Solution(tm). It's intended to buy the admin some time to solve the Real Issue. 2) Having a failsafe on standby such as this will save an admin's butt when it's 2am and PagerDuty won't shut up, and you're just awake enough to apply a temp fix and work on it in the morning. 3) Because "FIX…

Real Solutions (tm) are indeed nice, but hackers get shit done - this is an utterly shameless hack, and I do it myself.

Re: All my servers have an 8 GB empty file on disk

#242

For everyone saying "This isn't a real solution!" I'd like to explain why I think you're wrong. 1) It's not intended to be a Real Solution(tm). It's intended to buy the admin some time to solve the Real Issue. 2) Having a failsafe on standby such as this will save an admin's butt when it's 2am and PagerDuty won't shut up, and you're just awake enough to apply a temp fix and work on it in the morning. 3) Because "FIX…

I find that either a server needs more space, or has files that can be deleted. For the former you just increase the disk space, since most things are VMs these days and increasing space is easy. For the latter you can usually delete enough files to get the service back up before you start the proper cleanup.

If you really need some reserve space (physical server), I'd much rather store it in a vg (or zfs/btrfs subvolume). Will you remember the file exists at 2am? What about the other admins on your team?

Re: All my servers have an 8 GB empty file on disk

#243
post #79

In the mid nineties I worked in a research institute. There was a large shared Novell drive which was always on the verge of full. Almost every day we were asked to clean up our files as much as possible. There were no disc quota for some reason. One day I was working with my colleague and when the fileserver was full he went to a project folder and removed a file called balloon.txt which immediately freed up a few p…

At the opposite end, I heard a story of actually full storage from the beginning of the century, when I worked at a "large medical and research institution in the Midwest". They had expensive SMB shares (NetApp?) that kept getting full all the time. So they did the sane thing in the era of Napster: they started deleting MP3 files, with or without prior warning. Pretty soon, they got an angry call that music could not be played in the operating room. Oops. Surgeons, as you can guess, were treated like royalty and didn't appreciate seeing their routines disrupted.

Re: All my servers have an 8 GB empty file on disk

#244

A lot of tips in this thread are about how to better alert when you get low on disk space, how to recover, etc. but I'd like to highlight the statement: "The disk filled up, and that's one thing you don't want on a Linux server—or a Mac for that matter. When the disk is full nothing good happens." As developers, we need to be better at handling edge cases like out of disk space, out of memory, pegged bandwidth and pe…

I know when our server's /tmp directory is full because Bash's tab autocompletion stops working.

/home still has space, though, so nothing truly breaks. Perhaps I should file a bug report about that.

Re: All my servers have an 8 GB empty file on disk

#245

Earlier quoted context omitted.

always fun when you're barreling down the highway and the engine starts to lean out, prompting you to hurriedly locate and switch the petcock over before the engine stalls completely. suppose then that you go fill up and forget to set the petcock back to normal. 8ball says: "I see a long walk in your future."

Surprised there isn't a mechanism that mechanically switches the petcock over when you put a fuel nozzle up to the port

Typically there aren't two separate tanks - In one tank there are two tubes at different heights. As the fuel level falls below the height of the "main" tube the engine sputters, then turning the petcock engages the lower down "reserve" tube which is still below the fuel level. It's more of a warning than a true reserve, and most bikes with an actual fuel gauge don't have a reserve.

Re: All my servers have an 8 GB empty file on disk

#246

Earlier quoted context omitted.

What?

If it's morally incorrect to get to the license server early and snatch a roaming license you might not use to its fullest, is it not also wrong to access the property market early and snatch a bunch of land you might not use to its fullest?

It is unhealthy group behaviour. Your actions reduces the groups access. By sharing you increase group access and group success.

When you buy a property for speculation you take a risk on your ability to increase market value (or outside reasons). You could lose money.

Hoarding group resources makes you a net negative in your group. Hoarding property alone means you had to create or borrow enough value to obtain the property and finance upkeep. Someone has the money to build a new home. That transaction is a net positive.

Re: All my servers have an 8 GB empty file on disk

#247

This won't work with ZFS, as it may be impossible to delete a file on ZFS when disk is full. The equivalent in ZFS is to create an empty dataset with reserved space.

For me, it was possible to truncate -s 0 a large file on a full disk with ZFS.

Re: All my servers have an 8 GB empty file on disk

#248
post #69

Showing off that i’m not a sysadmin, but wouldn’t a monitoring daemon work? Once disk usage grows past a certain uncomfortable threshold you get an email/notification to see what’s up. I mean you obviously are monitoring other server vitals anyway right?

Cases mentioned below where space fills up quickly due to a bug, maybe yes. Outside that there's the problem that you can ignore the emails (or be sick, asleep, etc). Worse if they go to a team and everyone is busy and assumes someone else will deal. Bad if you aren't in charge and tell people in charge and they nod and don't decide anything - they prefer to run at 70/80/90/95% used indefinitely instead of signing a cheque.

When the drive fills and everything breaks and, you /have/ to respond, and it becomes socially ok to make it your highest priority and drop everything. An email "only a few weeks until maybe it runs out of space" is much harder to prioritise and get anyone to care. With this system, the time when it fills and breaks has some flex for you to not go down with the server, and save your own bacon. It's as much a fix for the organization as anything else.

I see this most in smaller company aging systems where they had ample storage and drives for a few years ago when they were new, now they're crammed full with the growth of software, libraries, updates, data, new services being deployed on them, increased demands, and nobody wants to commit to more storage for an older system towards the end of its warranties, but they definitely don't want to commit to the much larger cost of a replacement and all the project work, and running at 90% full costs nothing and involves no decisions. 91%. 92%.

Re: All my servers have an 8 GB empty file on disk

#249
post #54

This points to a much more serious problem. This is 2021 and the technology is from the 90s, with a really poor user experience design. Your car warns you when you're low on fuel, but your server doesn't if you're low on critical resources.

Alerting is also a hack really. In 2021 the operating SYSTEM should work as a system - complexly managing it's resources and make intelligent decisions. Ideally OS should dynamically reserve as much resources as needed on it's own.

Re: All my servers have an 8 GB empty file on disk

#250

> On Linux servers it can be incredibly difficult for any process to succeed if the disk is full. Copy commands and even deletions can fail or take forever as memory tries to swap to a full disk and there's very little you can do to free up large chunks of space. This reasoning doesn't make sense. On Linux, swap is preallocated. This is true regardless of whether you're using a swap partition or a swap file. See man…

IIRC, swap is actually needed for some memory operations. And when you run out of memory, the behaviour is often worse without swap.

These days I always at least configure an in-memory compressed swap (zram).

Post reply on HN