Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Running out of disk space in production

alt-romes.github.io

Post reply on HN
Print view View original

61–70 of 141 posts

Re: Running out of disk space in production

#61

Post by justsomehnguy »

dspillett wrote: post #15
flanfly wrote: post #2

A neat trick I was told is to always have ballast files on your systems. Just a few GiB of zeros that you can delete in cases like this. This won't fix the problem, but will buy you time and free space for stuff like lock files so you can get a working system.

Similarly, I always leave some space unallocated on LMV volume groups. It means that I can temporarily expand a volume easily if needed. It also serves to leave some space unused to help out the wear-levelling on the SSDs on which the RAID array that is the PV¹ for LVM. I'm, not 100% sure this is needed any more² but I've not looked into that sufficiently so until I do I'll keep the habit. -------- [1] if there are m…

Not needed. All your unused/unfilled space is that space for wear-leveling. It wasn't needed even back then besides some corner cases. And most importantly 10% of the drive in ~2010 were 6-12GB, nowadays it's 50-100GB at least.

Re: Running out of disk space in production

#63

Post by __david__ »

fragmede wrote: post #55
ape4 wrote: post #30

Earlier quoted context omitted.

If I recall correctly: dd if=/dev/urandom of=/home/myrandomfile bs=1 count=N

bs=1 is a recipe for waiting far longer than you have to because of the overhead of the system calls. Better bs=N count=1

That’s also not great if you’re trying to make a 10 gigabyte file. In that case, use bs=1M and count=SizeInMB.

Re: Running out of disk space in production

#64

Post by SoftTalker »

I've run into that "process still has deleted files open" situation a few times. df shows disk full, but du can't account for all of it, that's your clue to run lsof and look for "deleted" files that are open.

Even more confusing can be cases where a file is opened, deleted or renamed without being closed, and then a different file is created under the orginal path. To quote the man page, "lsof reports only the path by which the file was opened, not its possibly different final path."

Re: Running out of disk space in production

#65

Post by prmoustache »

happycrappy wrote: post #50
flanfly wrote: post #2

A neat trick I was told is to always have ballast files on your systems. Just a few GiB of zeros that you can delete in cases like this. This won't fix the problem, but will buy you time and free space for stuff like lock files so you can get a working system.

Interesting strategy, can't believe I've never heard of this one before. Would it be more pragmatic to allocate a swap file instead? Something that provides a theoretical benefit in the short term vs a static reservation.

Because adding swap file is instantaneous, removing one that is in use can take a longtime unless you reboot the OS so you can't just nuke it quickly.

Re: Running out of disk space in production

#66

Post by Twirrim »

ape4 wrote: post #30
layer8 wrote: post #25

Earlier quoted context omitted.

Better fill those files with random bytes, to ensure the filesystem doesn’t apply some “I don’t actually have to store all-zero blocks” sparse-file optimization. To my knowledge no non-compressing file system currently does this, but who knows about the future.

If I recall correctly: dd if=/dev/urandom of=/home/myrandomfile bs=1 count=N

If you want to do it really quickly

    openssl enc -aes-256-ctr -pbkdf2 -pass pass:"$(date '+%s')"
Almost all CPUs have AES native instructions so you'll be able to produce pseudorandom junk really fast. Even my old system will produce it at about 3Gb/s. Much faster than urandom can go.

Re: Running out of disk space in production

#67

Post by marcosdumay »

__david__ wrote: post #63
fragmede wrote: post #55

Earlier quoted context omitted.

bs=1 is a recipe for waiting far longer than you have to because of the overhead of the system calls. Better bs=N count=1

That’s also not great if you’re trying to make a 10 gigabyte file. In that case, use bs=1M and count=SizeInMB.

Modern computers are crazily overengineered...

Most current desktops (smaller than your usual server) won't have any problem with the GP's command. Yours is still better, of course.

Re: Running out of disk space in production

#68

Post by jasonpeacock »

flanfly wrote: post #2

A neat trick I was told is to always have ballast files on your systems. Just a few GiB of zeros that you can delete in cases like this. This won't fix the problem, but will buy you time and free space for stuff like lock files so you can get a working system.

> A neat trick I was told is to always have sleep statements in your code. Just a few sleep statements that you can delete in cases like this. This won't fix the problem, but will buy you time and free up latency for stuff like slow algorithms so you can get faster code.

FTFY ;)

Re: Running out of disk space in production

#69

Post by freedomben »

kccqzy wrote: post #39
freedomben wrote: post #35

Earlier quoted context omitted.

Yep, btrfs will happily do this to you. I verified it the hard way

Well btrfs supports compression so that’s understandable. However I personally prefer to control compression manually so it only compresses files marked by me for compression using chattr(1).

I've switched to that also. It surely wastes some space but being able to reason about file space is worth it to me for now

Re: Running out of disk space in production

#70

Post by klaushardt »

flanfly wrote: post #2

A neat trick I was told is to always have ballast files on your systems. Just a few GiB of zeros that you can delete in cases like this. This won't fix the problem, but will buy you time and free space for stuff like lock files so you can get a working system.

This is my snippet i used alot. In doubt when even rm wont work just reboot.

Disc Space Insurance File

    fallocate -l 8G /tmp/DELETE_IF_OUT_OF_SPACE.img
https://gist.github.com/klaushardt/9a5f6b0b078d28a23fd968f75...
Post reply on HN