Stories like this and my own past memories make me so happy to work somewhere big.
All my servers have an 8 GB empty file on disk
121–130 of 715 posts
Re: All my servers have an 8 GB empty file on disk
#122Earlier quoted context omitted.
No but the logfile writers are usually running as root AFAIK. And this is what tends to fill up the disk.
On systemd systems, logfiles are written to disk under the journald user, `systemd-journal`.
Is that true for all logfiles? I still have plenty of daemons (by default) writing directly to some file in /var/log eg EXIM, Apache, and the like. Also plenty of system stuff still write to files in that directory. And yes this is a machine that uses systemd.
Re: All my servers have an 8 GB empty file on disk
#123This won't work with ZFS, as it may be impossible to delete a file on ZFS when disk is full. The equivalent in ZFS is to create an empty dataset with reserved space.
Re: All my servers have an 8 GB empty file on disk
#124For everyone saying "This isn't a real solution!" I'd like to explain why I think you're wrong.
1) It's not intended to be a Real Solution(tm). It's intended to buy the admin some time to solve the Real Issue.
2) Having a failsafe on standby such as this will save an admin's butt when it's 2am and PagerDuty won't shut up, and you're just awake enough to apply a temp fix and work on it in the morning.
3) Because "FIX IT NOW OR ELSE" is a thing. Okay, sure. Null the file and then fill it with 7GB. Problem solved, for now. Everybody is happy and now I can work on the Real Problem: Bob won't stop hoarding spam.
That is all.
Re: All my servers have an 8 GB empty file on disk
#125[flagged]
Re: All my servers have an 8 GB empty file on disk
#126This points to a much more serious problem. This is 2021 and the technology is from the 90s, with a really poor user experience design. Your car warns you when you're low on fuel, but your server doesn't if you're low on critical resources.
Everyone has this kind of alerting set up, but that's not the point. The beauty of this solution is that it's dead simple and will never fail. Alerting can fail or be ignored. It's the same as old VW beetles which had a reserve gas tank. When you ran out of gas you opened a valve and you could limp to a gas station. Less likely to fail versus a 1950's era gauge that is telling you you're low. Also impossible to ignor…
It's the same as old VW beetles which had a reserve gas tank. When you ran out of gas you opened a valve and you could limp to a gas station
In scuba diving there used to be "J-valves". When you had 50 bar left in the tank they would cut out. Then you would pull to reenable your air and return to the surface. Unsurprisingly they are no longer popular.
Re: All my servers have an 8 GB empty file on disk
#127> even deletions can fail or take forever
> in a moment of full-disk crisis I can simply delete it and buy myself some critical time to debug and fix the problem
Uhh...
Re: All my servers have an 8 GB empty file on disk
#128In theory, this is a good idea, but doesn't protect you in all cases. I have had instances on a few of my application servers where an event happened that dumped GB's worth of log data to the log files in a matter of a couple of minutes and filled up the drive (Thanks fast SSDs!). If I employed the strategy in the article, it would have only bought me a couple of more minutes worth of time, if that!
Re: All my servers have an 8 GB empty file on disk
#129Earlier quoted context omitted.
Assuming we're talking about VMs (2021 etc.), for a SME is there any downside to giving 2TB of space to your discs and let dynamic allocation do the work? Perhaps consolidate/defrag once a year. Even monitoring total usage more often than that is probably not worth the effort - just buy ample cheap storage. Also, there was a tradition to split drives into OS, DB, DB Logs. That was mostly a rust performance thing and…
One VM using excessively more disk space than it's supposed to can potentially cause data corruption in all the other VMs on that system. For just spinning VMs up and down for testing, you probably won't run into that issue, but on a production system, it could potentially cause some massive downtime
Virtual machine disk space (e.g. Xen, Linode, AWS EC2, or similar) does not work this way. Each VM gets a dedicated amount of disk space allocated to it, they don't all share a pool of free space.
Re: All my servers have an 8 GB empty file on disk
#130How is this better than sounding alarms when free disk space drops below 8GB? If you’re going to ignore the alarms, then you’re going to have the same problem after you remove your spacer file and the disk fills up again!
[deleted]