Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Running out of disk space in production

alt-romes.github.io

Post reply on HN
Print view View original

51–60 of 141 posts

Re: Running out of disk space in production

#51

Post by throw0101d »

throw0101d wrote: post #22
flanfly wrote: post #2

A neat trick I was told is to always have ballast files on your systems. Just a few GiB of zeros that you can delete in cases like this. This won't fix the problem, but will buy you time and free space for stuff like lock files so you can get a working system.

> A neat trick I was told is to always have ballast files on your systems. ZFS has a "reservation" mechanism that's handy: > The minimum amount of space guaranteed to a dataset, not including its descendants. When the amount of space used is below this value, the dataset is treated as if it were taking up the amount of space specified by refreservation. The refreservation reservation is accounted for in the parent da…

Typo; link should be:

* https://openzfs.github.io/openzfs-docs/man/master/7/zfsprops...

Addendum: there's also the built-in compression functionality:

> When set to on (the default), indicates that the current default compression algorithm should be used. The default balances compression and decompression speed, with compression ratio and is expected to work well on a wide variety of workloads. Unlike all other settings for this property, on does not select a fixed compression type. As new compression algorithms are added to ZFS and enabled on a pool, the default compression algorithm may change. The current default compression algorithm is either lzjb or, if the lz4_compress feature is enabled, lz4.

* https://openzfs.github.io/openzfs-docs/man/master/7/zfsprops...

Re: Running out of disk space in production

#52

Post by grugdev42 »

You missed out point five.

5. Implement infrastructure monitoring.

Assuming you're on something like Ubuntu, the monit program is brilliant.

It's open source and self hosted, configured using plain text files, and can run scripts when thresholds are met.

I personally have it configured to hit a Slack webhook for a monitoring channel. Instant notifications for free!

Re: Running out of disk space in production

#55

Post by fragmede »

ape4 wrote: post #30
layer8 wrote: post #25

Earlier quoted context omitted.

Better fill those files with random bytes, to ensure the filesystem doesn’t apply some “I don’t actually have to store all-zero blocks” sparse-file optimization. To my knowledge no non-compressing file system currently does this, but who knows about the future.

If I recall correctly: dd if=/dev/urandom of=/home/myrandomfile bs=1 count=N

bs=1 is a recipe for waiting far longer than you have to because of the overhead of the system calls. Better bs=N count=1

Re: Running out of disk space in production

#56

Post by SAI_Peregrinus »

6031769 wrote: post #46
bombcar wrote: post #19

Earlier quoted context omitted.

Some filesystems can be unable to delete a file if full. Something to be a bit worried about.

Please name and shame those filesystems so that we will all be forewarned.

Any Copy-on-Write filesystem can run into this. There's always some way around it, but it can be problematic if you only have one device, can't remember the steps to fix a full filesystem, and can't look up the steps because you can't launch a browser without it trying to make some files!

Re: Running out of disk space in production

#59

Post by ilaksh »

I'm not sure that his problems are really over if a LOT of people were downloading a 2GB file. It would depend on the plan. Especially if his server is in the US.

But maybe the European Hetzner servers still have really big limits even for small ones.

But still, if people keep downloading, that could add up.

Re: Running out of disk space in production

#60

Post by evil-olive »

omarqureshi wrote: post #4
flanfly wrote: post #2

A neat trick I was told is to always have ballast files on your systems. Just a few GiB of zeros that you can delete in cases like this. This won't fix the problem, but will buy you time and free space for stuff like lock files so you can get a working system.

Surely a 50% warning alarm on disk usage covers this without manual intervention?

> Surely a 50% warning alarm on disk usage covers this without manual intervention?

surely you don't need a fire extinguisher in your kitchen, if you have a smoke detector?

a "warning alarm" is a terrible concept, in general. it's a perfect way to lead to alert fatigue.

over time, you're likely to have someone silence the alarm because there's some host sitting at 57% disk usage for totally normal reasons and they're tired of getting spammed about it.

even well-tuned alert rules (ones that predict growth over time rather than only looking at the current value) tend to be targeted towards catching relatively "slow" leaks of disk usage.

there is always the possibility for a "fast" disk space consumer to fill up the disk more quickly than your alerting system can bring it to your attention and you can fix it. at the extreme end, for example, a standard EBS volume has a throughput of 125mb/sec. something that saturates that limit will fill up 10gb of free space in 80 seconds.

Post reply on HN