221–230 of 715 posts
All my servers have an alarm when disk space goes above 70%. It sends an email every hour once the disk usage goes above 70%. Never had a server go down because of disk space issue after adopting this practise. Also one of the main reasons server disks go full is generally log files. Always remember to "logrotate" your log files and you will not have this issue that much. Yes one more thing, for all user uploaded fil…
The real question... Why does Linux or at least the common filesystems get stuck so easily running out of disk space? Surely normal commands like `rm` should still function.
> Surely normal commands like `rm` should still function They do. In my experience, the only disruption to most terminal operations is that tab completion will fail with an error.
> Copy commands and even deletions can fail
I've had that happen too many times, so I don't know why would fill up my disk with a hacky spacer file, which surely can also fail to be deleted when the disk is already full.
Earlier quoted context omitted.
I still know people that pre-emptively buy toilet paper "because hoarders might buy the rest of it" with absolutely no introspection.
Hoarders will buy the rest of it. During the 2-month lockdown a year ago, I would purchase 4 frozen pizzas at a time when I had a chance to buy them, because I was so upset that I could not buy one when I wanted a single one during the 2 previous weeks, because of hoarders who had been faster than me. People think of toilet papers, but it is not just that. Pasta, rice, flour, yeast, plenty of useful things went missi…
This is like carrying around a pound of beef because you refuse to look up the address of a McDonald's 7 minutes away. Setup quotas or implement some damn monitoring -- if you're not monitoring something as simple and critical as disk usage, what else are you not monitoring?
Quotas are tricky to set up when things are sharing disk space, and that could easily give you a false positive where a service unnecessarily runs out of space.
A lot of tips in this thread are about how to better alert when you get low on disk space, how to recover, etc. but I'd like to highlight the statement: "The disk filled up, and that's one thing you don't want on a Linux server—or a Mac for that matter. When the disk is full nothing good happens." As developers, we need to be better at handling edge cases like out of disk space, out of memory, pegged bandwidth and pe…
"I have an unlimited data plan, I'll just store everything in the cloud." only to discover later that unlimited has an asterisk by it and a footnote that says "LOL it's still limited".
This is a low cost way to make failure of your first line of defense less painful to recover, and seems like a Good Idea for those managing bare-metal non-cattle systems.
Earlier quoted context omitted.
I still know people that pre-emptively buy toilet paper "because hoarders might buy the rest of it" with absolutely no introspection.
Hoarders will buy the rest of it. During the 2-month lockdown a year ago, I would purchase 4 frozen pizzas at a time when I had a chance to buy them, because I was so upset that I could not buy one when I wanted a single one during the 2 previous weeks, because of hoarders who had been faster than me. People think of toilet papers, but it is not just that. Pasta, rice, flour, yeast, plenty of useful things went missi…
I think I "prepped" for the worst by buying a 10 lb bag of flour, 40 lb of rice, 5 lbs of oatmeal, a sack of potatoes and a few bags of beef jerky and trail mix. Even if the worst didn't come to pass I figured we'd eventually eat it all anyways and it wouldn't really be hoarded, but in pinch we could ration it and it would last a few months and give us nutritious meals. It's basically what ships crews used to survive on during the age of sail as they spent months at sea. Not a ton of variety but it will keep you alive.
IIRC 5% is reserved when the filesystem os created, and if it gets full you can run:
tune2fs -m 4 /dev/whatever
which will instantly make 1% of the disk available.
of course should be used sparingly and restored when finished
Same idea as this game development legend https://www.dodgycoder.net/2012/02/coding-tricks-of-game-dev... > he had put aside those two megabytes of memory early in the development cycle. He knew from experience that it was always impossible to cut content down to memory budgets, and that many projects had come close to failing because of it. So now, as a regular practice, he always put aside a nice block of memory to…
There's a difference between "The server is not responding right now. We're loosing customers.", and "Low resources during product development". Actually the latter may be a case of enforcing premature optimization. So no, it's not the same idea.