Running out of disk space in production
alt-romes.github.io
Running out of disk space in production
1–10 of 141 posts
Re: Running out of disk space in production
#2A neat trick I was told is to always have ballast files on your systems. Just a few GiB of zeros that you can delete in cases like this. This won't fix the problem, but will buy you time and free space for stuff like lock files so you can get a working system.
Re: Running out of disk space in production
#3A neat trick I was told is to always have ballast files on your systems. Just a few GiB of zeros that you can delete in cases like this. This won't fix the problem, but will buy you time and free space for stuff like lock files so you can get a working system.
Love the simplicity and pragmatism of this solution
Re: Running out of disk space in production
#4A neat trick I was told is to always have ballast files on your systems. Just a few GiB of zeros that you can delete in cases like this. This won't fix the problem, but will buy you time and free space for stuff like lock files so you can get a working system.
Surely a 50% warning alarm on disk usage covers this without manual intervention?
Re: Running out of disk space in production
#5A neat trick I was told is to always have ballast files on your systems. Just a few GiB of zeros that you can delete in cases like this. This won't fix the problem, but will buy you time and free space for stuff like lock files so you can get a working system.
Surely a 50% warning alarm on disk usage covers this without manual intervention?
If the alarms are reliably configured, confirmed to be working, low noise enough to be actioned, etc etc.
And of course there's nothing to say that both of these things can't be done simultaneously.
Re: Running out of disk space in production
#6A neat trick I was told is to always have ballast files on your systems. Just a few GiB of zeros that you can delete in cases like this. This won't fix the problem, but will buy you time and free space for stuff like lock files so you can get a working system.
Love the simplicity and pragmatism of this solution
[dead]
Re: Running out of disk space in production
#7A neat trick I was told is to always have ballast files on your systems. Just a few GiB of zeros that you can delete in cases like this. This won't fix the problem, but will buy you time and free space for stuff like lock files so you can get a working system.
I did this too, but i also zipped the file, turns out it had great packing ratio!
Re: Running out of disk space in production
#8A neat trick I was told is to always have ballast files on your systems. Just a few GiB of zeros that you can delete in cases like this. This won't fix the problem, but will buy you time and free space for stuff like lock files so you can get a working system.
I did this too, but i also zipped the file, turns out it had great packing ratio!
Personally I just keep the file on a ramdisk so you can avoid having to fetch it from slow storage
Re: Running out of disk space in production
#9A neat trick I was told is to always have ballast files on your systems. Just a few GiB of zeros that you can delete in cases like this. This won't fix the problem, but will buy you time and free space for stuff like lock files so you can get a working system.
Surely a 50% warning alarm on disk usage covers this without manual intervention?
Depends. A Kubernetes container might have only a few megabytes of disk space, because it shouldn't need it.
Except that one time when .NET decides that the incoming POST is over some magic limit and it doesn't do the processing in-memory like before, but instead has to write it to disk, crashing the whole pod. Fun times.
Also my Unraid NAS has two drives in "WARNING! 98% USED" alert state. One has 200GB of free space, the other 330GB. Percentages in integers don't work when the starting number is too big :)
Re: Running out of disk space in production
#10A neat trick I was told is to always have ballast files on your systems. Just a few GiB of zeros that you can delete in cases like this. This won't fix the problem, but will buy you time and free space for stuff like lock files so you can get a working system.
Surely a 50% warning alarm on disk usage covers this without manual intervention?
If the alarm works. And it actioned not just snoozed too much or just dismissed entirely.
Defence in depth is a good idea: proper alarms, and a secondary measure in case they don't have the intended effect.