Live data from Hacker News

Running out of disk space in production

alt-romes.github.io

31–40 of 141 posts

Re: Running out of disk space in production

#31

> I rushed to run du -sh on everything I could, as that’s as good as I could manage. I recently came across gdu (1) and have installed/used it on every machine since then. [1]: https://github.com/dundee/gdu

Have you used ncdu? I wonder how this compares.

Re: Running out of disk space in production

#33

> I rushed to run du -sh on everything I could, as that’s as good as I could manage. I recently came across gdu (1) and have installed/used it on every machine since then. [1]: https://github.com/dundee/gdu

I use dust for this, but gdu looks nice, I'll give it a try. Thanks for sharing.

Re: Running out of disk space in production

#34

Earlier quoted context omitted.

Surely a 50% warning alarm on disk usage covers this without manual intervention?

If the alarm works. And it actioned not just snoozed too much or just dismissed entirely. Defence in depth is a good idea: proper alarms, and a secondary measure in case they don't have the intended effect.

[dead]

Re: Running out of disk space in production

#35
post #25
post #2

A neat trick I was told is to always have ballast files on your systems. Just a few GiB of zeros that you can delete in cases like this. This won't fix the problem, but will buy you time and free space for stuff like lock files so you can get a working system.

Better fill those files with random bytes, to ensure the filesystem doesn’t apply some “I don’t actually have to store all-zero blocks” sparse-file optimization. To my knowledge no non-compressing file system currently does this, but who knows about the future.

Yep, btrfs will happily do this to you. I verified it the hard way

Re: Running out of disk space in production

#37
post #2

A neat trick I was told is to always have ballast files on your systems. Just a few GiB of zeros that you can delete in cases like this. This won't fix the problem, but will buy you time and free space for stuff like lock files so you can get a working system.

I did this recently, aka, docker images prune. Can confirm, saved the day.

Re: Running out of disk space in production

#38
post #2

A neat trick I was told is to always have ballast files on your systems. Just a few GiB of zeros that you can delete in cases like this. This won't fix the problem, but will buy you time and free space for stuff like lock files so you can get a working system.

Surely a 50% warning alarm on disk usage covers this without manual intervention?

You don't want an alarm on a usage threshold, you want a linear regression that predicts when utilization will cross a threshold. Then you set your alarms for "How long does it take me to remediate this condition?"

Re: Running out of disk space in production

#39
post #25

Earlier quoted context omitted.

Better fill those files with random bytes, to ensure the filesystem doesn’t apply some “I don’t actually have to store all-zero blocks” sparse-file optimization. To my knowledge no non-compressing file system currently does this, but who knows about the future.

Yep, btrfs will happily do this to you. I verified it the hard way

Well btrfs supports compression so that’s understandable. However I personally prefer to control compression manually so it only compresses files marked by me for compression using chattr(1).

Re: Running out of disk space in production

#40
post #2

A neat trick I was told is to always have ballast files on your systems. Just a few GiB of zeros that you can delete in cases like this. This won't fix the problem, but will buy you time and free space for stuff like lock files so you can get a working system.

> A neat trick I was told is to always have ballast files on your systems. ZFS has a "reservation" mechanism that's handy: > The minimum amount of space guaranteed to a dataset, not including its descendants. When the amount of space used is below this value, the dataset is treated as if it were taking up the amount of space specified by refreservation. The refreservation reservation is accounted for in the parent da…

Also if you VMs on a disk backed by ZFS it's trivial to extend those disks provided you actually do have space on the real disk. (Even automatic with LXC).
Post reply on HN