A neat trick I was told is to always have ballast files on your systems. Just a few GiB of zeros that you can delete in cases like this. This won't fix the problem, but will buy you time and free space for stuff like lock files so you can get a working system.
Running out of disk space in production
11–20 of 141 posts
Re: Running out of disk space in production
#12A neat trick I was told is to always have ballast files on your systems. Just a few GiB of zeros that you can delete in cases like this. This won't fix the problem, but will buy you time and free space for stuff like lock files so you can get a working system.
Re: Running out of disk space in production
#13Earlier quoted context omitted.
Surely a 50% warning alarm on disk usage covers this without manual intervention?
If the alarm works. And it actioned not just snoozed too much or just dismissed entirely. Defence in depth is a good idea: proper alarms, and a secondary measure in case they don't have the intended effect.
Re: Running out of disk space in production
#14I recently came across gdu (1) and have installed/used it on every machine since then.
Re: Running out of disk space in production
#15A neat trick I was told is to always have ballast files on your systems. Just a few GiB of zeros that you can delete in cases like this. This won't fix the problem, but will buy you time and free space for stuff like lock files so you can get a working system.
It also serves to leave some space unused to help out the wear-levelling on the SSDs on which the RAID array that is the PV¹ for LVM. I'm, not 100% sure this is needed any more² but I've not looked into that sufficiently so until I do I'll keep the habit.
--------
[1] if there are multiple PVs, from different drives/arrays, in the VG, then you might need to manually skip a bit on each one because LVM will naturally fill one before using the next. Just allocate a small LV specially on each and don't use it. You can remove one/all of them and add the extents to the fill LV if/when needed. Giving it a useful name also reminds you why that bit of space is carved out.
[2] drives under-allocate by default IIRC
Re: Running out of disk space in production
#16A neat trick I was told is to always have ballast files on your systems. Just a few GiB of zeros that you can delete in cases like this. This won't fix the problem, but will buy you time and free space for stuff like lock files so you can get a working system.
Re: Running out of disk space in production
#17The author ended up doing this for /nix under pressure, but it's very much standard best practice in any unix/linux box, especially one with only 40GB.
Re: Running out of disk space in production
#18Re: Running out of disk space in production
#19A neat trick I was told is to always have ballast files on your systems. Just a few GiB of zeros that you can delete in cases like this. This won't fix the problem, but will buy you time and free space for stuff like lock files so you can get a working system.
Re: Running out of disk space in production
#20A neat trick I was told is to always have ballast files on your systems. Just a few GiB of zeros that you can delete in cases like this. This won't fix the problem, but will buy you time and free space for stuff like lock files so you can get a working system.
Would another way be to drop the reserved space (typically 1% to 5% on an ext file system)?