Live data from Hacker News

Running out of disk space in production

alt-romes.github.io

11–20 of 141 posts

Re: Running out of disk space in production

#11
post #2

A neat trick I was told is to always have ballast files on your systems. Just a few GiB of zeros that you can delete in cases like this. This won't fix the problem, but will buy you time and free space for stuff like lock files so you can get a working system.

This is why I never empty the Rubbish Bin/trash Can on my Linux laptop until the disk fills.

Re: Running out of disk space in production

#12
post #2

A neat trick I was told is to always have ballast files on your systems. Just a few GiB of zeros that you can delete in cases like this. This won't fix the problem, but will buy you time and free space for stuff like lock files so you can get a working system.

Would another way be to drop the reserved space (typically 1% to 5% on an ext file system)?

Re: Running out of disk space in production

#13

Earlier quoted context omitted.

Surely a 50% warning alarm on disk usage covers this without manual intervention?

If the alarm works. And it actioned not just snoozed too much or just dismissed entirely. Defence in depth is a good idea: proper alarms, and a secondary measure in case they don't have the intended effect.

Alarms are great, but when something goes wrong SSDs can fill up amazingly fast!

Re: Running out of disk space in production

#15
post #2

A neat trick I was told is to always have ballast files on your systems. Just a few GiB of zeros that you can delete in cases like this. This won't fix the problem, but will buy you time and free space for stuff like lock files so you can get a working system.

Similarly, I always leave some space unallocated on LMV volume groups. It means that I can temporarily expand a volume easily if needed.

It also serves to leave some space unused to help out the wear-levelling on the SSDs on which the RAID array that is the PV¹ for LVM. I'm, not 100% sure this is needed any more² but I've not looked into that sufficiently so until I do I'll keep the habit.

--------

[1] if there are multiple PVs, from different drives/arrays, in the VG, then you might need to manually skip a bit on each one because LVM will naturally fill one before using the next. Just allocate a small LV specially on each and don't use it. You can remove one/all of them and add the extents to the fill LV if/when needed. Giving it a useful name also reminds you why that bit of space is carved out.

[2] drives under-allocate by default IIRC

Re: Running out of disk space in production

#16
post #2

A neat trick I was told is to always have ballast files on your systems. Just a few GiB of zeros that you can delete in cases like this. This won't fix the problem, but will buy you time and free space for stuff like lock files so you can get a working system.

Similar to the old game development trick of hiding some memory away and then freeing it up near the end of development when the budget starts getting tight.

Re: Running out of disk space in production

#17
One thing that jumps out is the root filesystem, /nix/store, logs, temp files, and application data were all on the same partition. Putting /tmp, /var/log, and /nix on separate mount points (or at least using quotas) is a normal defense against exactly this kind of cascading failure. A runaway temp dir can't break your app ability to send outgoing emails.

The author ended up doing this for /nix under pressure, but it's very much standard best practice in any unix/linux box, especially one with only 40GB.

Re: Running out of disk space in production

#19
post #2

A neat trick I was told is to always have ballast files on your systems. Just a few GiB of zeros that you can delete in cases like this. This won't fix the problem, but will buy you time and free space for stuff like lock files so you can get a working system.

Some filesystems can be unable to delete a file if full. Something to be a bit worried about.

Re: Running out of disk space in production

#20
post #2

A neat trick I was told is to always have ballast files on your systems. Just a few GiB of zeros that you can delete in cases like this. This won't fix the problem, but will buy you time and free space for stuff like lock files so you can get a working system.

Would another way be to drop the reserved space (typically 1% to 5% on an ext file system)?

Reserved space doesn't protect you against root, who is often the user to blame for the last used MB.
Post reply on HN