201–210 of 715 posts
It sounds 'cool' and all for 1995.. but, what about one script that'll email you when the disk is at 80%?
So that if by the time you get the email the issue is at 97% you can immediately give yourself enough breathing room to figure things out with downtime or significantly degraded performance.
Earlier quoted context omitted.
No but the logfile writers are usually running as root AFAIK. And this is what tends to fill up the disk.
On systemd systems, logfiles are written to disk under the journald user, `systemd-journal`.
Earlier quoted context omitted.
Compare the rate an haywire process can fill up disk to your response time to alarms, and you got your answer right there.
Okay, so now you have a disk full, only become aware of it when it's full and your database throws errors. You have an easy way to fix it, just delete the spacer file. But what good does that do? You're still in the mess where your database is really unhappy. On the other hand, if your monitoring was set up well, you got a notification and had time to react to it before it was at 100%. Granted, if you have a process…
You solve the issue right then and there. Step 1. realize there is a space issue and get to terminal Step 2. free space so any solution has memory Step 3. Solve by doing
This reminds me of Perl’s esoteric $^M variable. You assign it some giant string, and in an out-of-memory condition, the value is cleared to free up some emergency space for graceful shutdown. “To discourage casual use of this advanced feature, there is no English long name for this variable.” But the language-build flag to enable it has a great name: -DPERL_EMERGENCY_SBRK, obviously inspired by emergency brake.
Earlier quoted context omitted.
Exactly, it's 1990s 'cool' - the time it took him to write the blog, he could have written a script that would send him updates on all his devices...
There's no reason not to have multiple fail-safes. Receiving the alert on a device at 3am would still mean he could free up 8gb immediately and have breathing room to solve the problem. And remember this is for a single admin. Asking such a person to be on call 24-7 all year, vacations, holidays, weekends... Having a quick way to get breathing room can significantly reduce the stress & cognitive load of worrying abou…
Because really the biggest problem when I've had a partition get full, is I sometimes can't even delete the offending log file.
For everyone saying "This isn't a real solution!" I'd like to explain why I think you're wrong. 1) It's not intended to be a Real Solution(tm). It's intended to buy the admin some time to solve the Real Issue. 2) Having a failsafe on standby such as this will save an admin's butt when it's 2am and PagerDuty won't shut up, and you're just awake enough to apply a temp fix and work on it in the morning. 3) Because "FIX…
> 1) It's not intended to be a Real Solution(tm). It's intended to buy the admin some time to solve the Real Issue. If you don't have monitoring, will you even be aware that your disk is filling up? If you do have monitoring, why are you artificially filling up your disk so that it will be at 100% more quickly instead of just setting your monitoring up to alert you when it's at $whateverItWasSetToMinusEightGB?
A second argument is it's not opened by any process. One problem I've had fixing disk full errors was figuring out which process still had a file open.
(For any POSIX noobs: the space occupied by a file is controlled by its inode. Deleting a file "unlinks" the inode from the directory, but an open filehandle counts as a link to that inode. Until all links to the inode are deleted, the OS won't release the space occupied by the file. Particularly with log files, you need to kill any processes that have it open to actually reclaim the disk space.)
Earlier quoted context omitted.
I still know people that pre-emptively buy toilet paper "because hoarders might buy the rest of it" with absolutely no introspection.
Hoarders will buy the rest of it. During the 2-month lockdown a year ago, I would purchase 4 frozen pizzas at a time when I had a chance to buy them, because I was so upset that I could not buy one when I wanted a single one during the 2 previous weeks, because of hoarders who had been faster than me. People think of toilet papers, but it is not just that. Pasta, rice, flour, yeast, plenty of useful things went missi…
Earlier quoted context omitted.
An extra failsafe? You can do both. What if your cron/netdata are not forwarding emails for some reason (eg nullmailer gets errors from Mailgun)?
Right, but again, what good does the spacer file do if you're not aware that you're running low on disk space? That is: if your monitoring isn't working, how do you know that you need to quickly make room? And if your monitoring is working correctly, the spacer file really serves no purpose other than lowering the available disk space.
2. Maybe you didn't get it, but "nullmailer not forwarding cron email due to mailgun problems" was a bit too specific to be an example I just made up, wasn't it? Again, the premise "if your monitoring is working correctly" is not a good one to base your reasoning upon. Especially if you have 1 VM (VPS) and not a whole k8s cluster with a devops team with rotational on-call assignments.