Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Running out of disk space in production

alt-romes.github.io

Post reply on HN
Print view View original

71–80 of 141 posts

Re: Running out of disk space in production

#71

Post by nyrikki »

layer8 wrote: post #25
flanfly wrote: post #2

A neat trick I was told is to always have ballast files on your systems. Just a few GiB of zeros that you can delete in cases like this. This won't fix the problem, but will buy you time and free space for stuff like lock files so you can get a working system.

Better fill those files with random bytes, to ensure the filesystem doesn’t apply some “I don’t actually have to store all-zero blocks” sparse-file optimization. To my knowledge no non-compressing file system currently does this, but who knows about the future.

XFS, Ext4, btrfs etc… all support sparse files, so any app can cause problems you can try it with:

    dd if=/dev/zero of=sparse_file.img bs=1M count=0 seek=1024
If you add conv=sparse to the dd command with a smaller block size it will sparsify what you copy too, use the wrong cp command flags and they will explode.

Much harder problem than the file system layers to deal with because the stat size will look smaller usually.

Re: Running out of disk space in production

#72

Post by ape4 »

Twirrim wrote: post #66
ape4 wrote: post #30

Earlier quoted context omitted.

If I recall correctly: dd if=/dev/urandom of=/home/myrandomfile bs=1 count=N

If you want to do it really quickly openssl enc -aes-256-ctr -pbkdf2 -pass pass:"$(date '+%s')" Almost all CPUs have AES native instructions so you'll be able to produce pseudorandom junk really fast. Even my old system will produce it at about 3Gb/s. Much faster than urandom can go.

That's very cool. Sadly running that exact command gets an incomplete file and error "error writing output file". It suggests adding iflag=fullblock (to dd). Running that makes a file of the correct size. But still gives "error writing output file". I suppose that occurs because dd breaks the pipe.

Re: Running out of disk space in production

#73

Post by justin_oaks »

RALaBarge wrote: post #47

Wait until you run out of inodes!

That happened to me exactly once in my 20-year career. It was on a web server (maybe even NGINX) that had too many cached files.

Even though it only happened once, I still set up monitoring for inode exhaustion.

Re: Running out of disk space in production

#74

Post by nyrikki »

renatovico wrote: post #57

Why not implement x send file ?

Came here to say this

X-Accel-Redirect (Nginx sendfile), if supported by Haskell is the way, it is zero copy and will dramatically help in many cases.

If you are modifying the body is one of the cases where it doesn’t work.

Re: Running out of disk space in production

#75

Post by lanstin »

RALaBarge wrote: post #47

Wait until you run out of inodes!

Old war story: I had an old Sun 4/260 with 2 1G drives - I had SunOS on 1 and Gentoo on the other - my initial Gentoo install worked for a while but then the portage directory used all the configured iNodes - really weird errors and I could not figure it out at the time; error msgs maybe should mention inodes? I had to do #gentoo-sun IRC and someone suggested df -i which was indeed the issues (solve: you can configure extN filesystems to have more iNodes)

Re: Running out of disk space in production

#76

Post by layer8 »

nyrikki wrote: post #71
layer8 wrote: post #25

Earlier quoted context omitted.

Better fill those files with random bytes, to ensure the filesystem doesn’t apply some “I don’t actually have to store all-zero blocks” sparse-file optimization. To my knowledge no non-compressing file system currently does this, but who knows about the future.

XFS, Ext4, btrfs etc… all support sparse files, so any app can cause problems you can try it with: dd if=/dev/zero of=sparse_file.img bs=1M count=0 seek=1024 If you add conv=sparse to the dd command with a smaller block size it will sparsify what you copy too, use the wrong cp command flags and they will explode. Much harder problem than the file system layers to deal with because the stat size will look smaller usua…

Creating sparse files requires the application to purposefully use special calls like fallocate() or seek beyond EOF, like dd with conv=sparse does. You won't accidentally create a sparse file just by filling a file with zeros.

Re: Running out of disk space in production

#77

Post by nyrikki »

layer8 wrote: post #76
nyrikki wrote: post #71

Earlier quoted context omitted.

XFS, Ext4, btrfs etc… all support sparse files, so any app can cause problems you can try it with: dd if=/dev/zero of=sparse_file.img bs=1M count=0 seek=1024 If you add conv=sparse to the dd command with a smaller block size it will sparsify what you copy too, use the wrong cp command flags and they will explode. Much harder problem than the file system layers to deal with because the stat size will look smaller usua…

Creating sparse files requires the application to purposefully use special calls like fallocate() or seek beyond EOF, like dd with conv=sparse does. You won't accidentally create a sparse file just by filling a file with zeros.

It is an observability issue, even zabbix tracked reserve space and inodes 20 years ago.

Will dedupe,compression,sparse files you simply don’t track utilization by clients view, which is what du does.

The concrete implementation is what matters and what is, as this case demonstrates, is what you should alert on.

Inodes, blocks, extents etc.. are what matters, not the user view of data size.

Even with rrdtool you could set reasonable alerts, but the heuristics of someone exploding a sparse file with a non-sparse copy makes that harder.

Rsync ssh etc… will do that by default.

Re: Running out of disk space in production

#79

Post by seabrookmx »

entropie wrote: post #14

> I rushed to run du -sh on everything I could, as that’s as good as I could manage. I recently came across gdu (1) and have installed/used it on every machine since then. [1]: https://github.com/dundee/gdu

I can't recommend `dust` enough: https://github.com/bootandy/dust

Re: Running out of disk space in production

#80

Post by rubatuga »

dizhn wrote: post #40
throw0101d wrote: post #22

Earlier quoted context omitted.

> A neat trick I was told is to always have ballast files on your systems. ZFS has a "reservation" mechanism that's handy: > The minimum amount of space guaranteed to a dataset, not including its descendants. When the amount of space used is below this value, the dataset is treated as if it were taking up the amount of space specified by refreservation. The refreservation reservation is accounted for in the parent da…

Also if you VMs on a disk backed by ZFS it's trivial to extend those disks provided you actually do have space on the real disk. (Even automatic with LXC).

Please explain!
Post reply on HN