[flagged]
Not really.
All my servers have an 8 GB empty file on disk
161–170 of 715 posts
Re: All my servers have an 8 GB empty file on disk
#162Earlier quoted context omitted.
I know about this, but I do think it's not a bad idea doing what he does because the reserved block count is for root and most server processes still run as root. And it's usually them that are causing the disk to fill. Though I suppose this also makes the problem itself more prominent in the first place. I guess if you run into this a lot, stricter monitoring would be a better solution. The way I found out about it…
What servers usually run as root? Some may start as root, but usually drop privileges for the actual server processes quickly, eg. apache, nginx, sshd. Nothing that actually does the "serving" or accesses data should be running as root.
It used to be common, before "the cloud", to have many apparently unnecessary partitions in a server install. One for /, one for /var, one for /home, one for swap at the low sector numbers...
The idea is that /var filling up would not make the system unrecoverable.
Re: All my servers have an 8 GB empty file on disk
#163In the mid nineties I worked in a research institute. There was a large shared Novell drive which was always on the verge of full. Almost every day we were asked to clean up our files as much as possible. There were no disc quota for some reason. One day I was working with my colleague and when the fileserver was full he went to a project folder and removed a file called balloon.txt which immediately freed up a few p…
Perfect example of the tragedy of the commons. If individuals don't create these balloon files then they won't be able to use the file server when they need it, yet by creating these balloon files the collective action depletes the shared resource of its main function.
Re: All my servers have an 8 GB empty file on disk
#164For everyone saying "This isn't a real solution!" I'd like to explain why I think you're wrong. 1) It's not intended to be a Real Solution(tm). It's intended to buy the admin some time to solve the Real Issue. 2) Having a failsafe on standby such as this will save an admin's butt when it's 2am and PagerDuty won't shut up, and you're just awake enough to apply a temp fix and work on it in the morning. 3) Because "FIX…
Setup proper monitoring and never get to the Real Issue to begin with. These sysadmin hacks are not helpful
In a perfect world, this is true. But we don't have one of those.
Re: All my servers have an 8 GB empty file on disk
#165This points to a much more serious problem. This is 2021 and the technology is from the 90s, with a really poor user experience design. Your car warns you when you're low on fuel, but your server doesn't if you're low on critical resources.
Linux servers aren't like mass consumer products. It's assumed users know what they're doing and can build and configure what they need on top of it.
> This is 2021 and the technology is from the 90s I don't see how this is a valid point. Is integrated circuit technology outdated because it was developed in the 60s?
Re: All my servers have an 8 GB empty file on disk
#166The real question... Why does Linux or at least the common filesystems get stuck so easily running out of disk space? Surely normal commands like `rm` should still function.
Re: All my servers have an 8 GB empty file on disk
#167It's the google photos thumbnails db. /s
Re: All my servers have an 8 GB empty file on disk
#168My understanding is this is why one should partition a drive. If you have a data partition, a swap partition, and an OS partition, you can get around issues where a server’s lack of disk space hoses the whole system.
100% agree. I think at the bare minimum every system should have two partitions: `/` and `/var` /var is usually where the most data gets added. Logs, database files, caches, and whatever other junk your app spits out. 99% of the time that's what causes the out of space issues. By keeping that separate from root, you've saved your system from being completely hosed when it fills up (which it will). Obviously there are…
Sounds like a poor-mans quota system
Re: All my servers have an 8 GB empty file on disk
#169For everyone saying "This isn't a real solution!" I'd like to explain why I think you're wrong. 1) It's not intended to be a Real Solution(tm). It's intended to buy the admin some time to solve the Real Issue. 2) Having a failsafe on standby such as this will save an admin's butt when it's 2am and PagerDuty won't shut up, and you're just awake enough to apply a temp fix and work on it in the morning. 3) Because "FIX…
This reminds me of the reserve tank toggle on some motorcycles. When you run out of gas, you switch the toggle and drive directly to a gas station.
always fun when you're barreling down the highway and the engine starts to lean out, prompting you to hurriedly locate and switch the petcock over before the engine stalls completely.
suppose then that you go fill up and forget to set the petcock back to normal. 8ball says: "I see a long walk in your future."
Re: All my servers have an 8 GB empty file on disk
#170How is this better than sounding alarms when free disk space drops below 8GB? If you’re going to ignore the alarms, then you’re going to have the same problem after you remove your spacer file and the disk fills up again!
Compare the rate an haywire process can fill up disk to your response time to alarms, and you got your answer right there.
Okay, so now you have a disk full, only become aware of it when it's full and your database throws errors. You have an easy way to fix it, just delete the spacer file. But what good does that do? You're still in the mess where your database is really unhappy.
On the other hand, if your monitoring was set up well, you got a notification and had time to react to it before it was at 100%.
Granted, if you have a process that just wrote a file at maximum speed, that time window is tiny, but that's not usually what happens in my experience. What happens is that something starts logging more and it slowly builds up while you're happy that your server is running so well that you don't need to pay attention. And then the alert comes and tells you that there's less than 10% space available, and you have plenty of time to investigate and avert the crisis.