Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

All my servers have an 8 GB empty file on disk

brianschrader.com

Post reply on HN
Print view View original

211–220 of 715 posts

Re: All my servers have an 8 GB empty file on disk

#211

Post by vineyardmike »

copperfoil wrote: post #133
rozap wrote: post #118

Earlier quoted context omitted.

Everyone has this kind of alerting set up, but that's not the point. The beauty of this solution is that it's dead simple and will never fail. Alerting can fail or be ignored. It's the same as old VW beetles which had a reserve gas tank. When you ran out of gas you opened a valve and you could limp to a gas station. Less likely to fail versus a 1950's era gauge that is telling you you're low. Also impossible to ignor…

> The beauty of this solution is that it's dead simple and will never fail. Alerting can fail or be ignored. It's not that straightforward IMO. Would this file be deleted before the space is filled? If so, there is alerting in place, and it assumes there's a way to delete files before space fills up. If this file is deleted after space fills up, how is this different from not having the file, other than making findin…

>you'd have to solve the problem the same way if you didn't use this method.

What if the solution required some amount of free space? (eg. installing a package or swap)

Re: All my servers have an 8 GB empty file on disk

#212

Post by bane »

rgj wrote: post #79

In the mid nineties I worked in a research institute. There was a large shared Novell drive which was always on the verge of full. Almost every day we were asked to clean up our files as much as possible. There were no disc quota for some reason. One day I was working with my colleague and when the fileserver was full he went to a project folder and removed a file called balloon.txt which immediately freed up a few p…

This is a surprisingly common hoarding behavior among humans using scarce resources. In technology you see it everywhere, virtualization infrastructure, disk storage, etc.

This is actually kind of clever. How the tribal knowledge for how to "reserve space" was developed and disseminated would be pretty interesting to study.

Re: All my servers have an 8 GB empty file on disk

#213

Post by jodrellblank »

mtone wrote: post #74
diego wrote: post #54

This points to a much more serious problem. This is 2021 and the technology is from the 90s, with a really poor user experience design. Your car warns you when you're low on fuel, but your server doesn't if you're low on critical resources.

Assuming we're talking about VMs (2021 etc.), for a SME is there any downside to giving 2TB of space to your discs and let dynamic allocation do the work? Perhaps consolidate/defrag once a year. Even monitoring total usage more often than that is probably not worth the effort - just buy ample cheap storage. Also, there was a tradition to split drives into OS, DB, DB Logs. That was mostly a rust performance thing and…

If you are using less space than the underlying datastore, there's no benefit to dynamic allocation, you may as well give the servers larger fixed disks. If you are thinking that one server might need more than the fixed size for a sudden growth, then you need to be monitoring to deal with that because that will run out of your space. If you are overprovisioning the datastore, you have the same problem at a level lower, and need to be monitoring that and alerting for that instead (as well).

> "just buy ample cheap storage"; "That was mostly a rust performance thing and these days is probably just"

In the UK a 6TB enterprise rust disk is £150 and a 2TB enterprise SSD is £300, it's 6x the price to SSD everything, and take 3x more drive bays so add more for that. And you can never "just" buy more storage than you ever need - apart from the obvious "when you bought it, you thought you were buying enough, because if you thought you needed more you would have bought more", so that amounts to saying "just know the future better", but it can't happen because Parkinson's Law ("work expands so as to fill the time available for its completion") applies to storage, the more there is available, the more things appear to fill it up.

Room for a test restore of the backups in that space. Room for a clone of the database to do some testing. Room for a trial of a new product. Room for a copy of all the installers and packagers for convenience. Room for a massive central logging server there. What do you mean it's full?

Re: All my servers have an 8 GB empty file on disk

#214

Post by michaelt »

dmingod666 wrote: post #138
legulere wrote: post #125

[flagged]

What?

If it's morally incorrect to get to the license server early and snatch a roaming license you might not use to its fullest, is it not also wrong to access the property market early and snatch a bunch of land you might not use to its fullest?

Re: All my servers have an 8 GB empty file on disk

#215

Post by npongratz »

nijave wrote: post #166

The real question... Why does Linux or at least the common filesystems get stuck so easily running out of disk space? Surely normal commands like `rm` should still function.

As recently as 2016 I experienced major problems using `rm` with an intentionally-filled btrfs (and current Linux kernel at the time), and per my notes, it was even mounted as `-o nodatacow`:

    # rm -f /mnt/data/zero.*.fill
    rm: cannot remove '/mnt/data/zero.1.fill': No space left on device

Re: All my servers have an 8 GB empty file on disk

#216

Post by luckylion »

bostonsre wrote: post #86
h4waii wrote: post #37

This is like carrying around a pound of beef because you refuse to look up the address of a McDonald's 7 minutes away. Setup quotas or implement some damn monitoring -- if you're not monitoring something as simple and critical as disk usage, what else are you not monitoring?

Not all environments require a stringent SLA. I have some servers that don't have a stringent SLA and aren't worth being woken up at night over if their disk is filling up fast.

Okay, so you wake up with a full disk. What did the spacer accomplish?

Re: All my servers have an 8 GB empty file on disk

#217

Post by quesera »

dnautics wrote: post #193
0_____0 wrote: post #169

Earlier quoted context omitted.

always fun when you're barreling down the highway and the engine starts to lean out, prompting you to hurriedly locate and switch the petcock over before the engine stalls completely. suppose then that you go fill up and forget to set the petcock back to normal. 8ball says: "I see a long walk in your future."

Surprised there isn't a mechanism that mechanically switches the petcock over when you put a fuel nozzle up to the port

Most motorcycles are surprisingly manual. This was originally a necessity (like in cars), but remains aesthetically preferable for many riders.

OTOH, Honda Goldwings have stereo systems. They might grow an automatic fuel reserve switcher-backer someday too. :)

Re: All my servers have an 8 GB empty file on disk

#218

Post by louwrentius »

Please use LVM (Logical Volume Manager) if you really are afraid of filling up disks.

If the disk would ever fill up:

    1. Buy an additional virtual disk
    2. Add the disk to the LVM volume group
    3. Expand the Logical volume
A really good primer on LVM:

https://wiki.archlinux.org/index.php/LVM

Re: All my servers have an 8 GB empty file on disk

#220

Post by pvorb »

luckylion wrote: post #187
smarx007 wrote: post #151

Earlier quoted context omitted.

An extra failsafe? You can do both. What if your cron/netdata are not forwarding emails for some reason (eg nullmailer gets errors from Mailgun)?

Right, but again, what good does the spacer file do if you're not aware that you're running low on disk space? That is: if your monitoring isn't working, how do you know that you need to quickly make room? And if your monitoring is working correctly, the spacer file really serves no purpose other than lowering the available disk space.

Somebody will notify you. If the service is just for yourself, you don't need monitoring at all.
Post reply on HN