Live data from Hacker News

Running out of disk space in production

alt-romes.github.io

131–140 of 141 posts

Re: Running out of disk space in production

#131
post #30

Earlier quoted context omitted.

If I recall correctly: dd if=/dev/urandom of=/home/myrandomfile bs=1 count=N

My choice has always been `shred`: $ sudo truncate --size 1G /emergency-space $ sudo shred /emergency-space I find it widely available, even in tiny distros.

Very cool. Reading the man page I see, by default, shred does 3 iterations. Adding --iterations=1 makes it random enough for this purpose and faster.

Re: Running out of disk space in production

#132

Earlier quoted context omitted.

> but enforcing a few percent extra helps. Only if you fill the drive up to 95-99% and do this often. Otherwise it's just a cargo-cult. > So while I would say to use a lower percentage as space increases If your drive is over-provisioned (eg 960GB instead of 1024GB) then it's not needed. If not and you fill your drive to the full and just want to be sure then you need the size of the biggest write you would do plus s…

Nah, we used some consumer SSD for write heavy but not all that precious data, and time to live was basically directly dependant on the space left free on device. Of course, doesn't matter for desktop use as the spare on drive is enough, but still, if you have 24/7 write heavy loads, making sure it's all trimmed will noticably extend lifetime

Yes, this is the reason why 0.3 DWPD drive is 10 orders smaller than 3 DWPD. I know the horror stories of using Samsung EVO for the SQL loads, especially But yes, without the actual use-case it's just speculations.

NB QVO drives I mentioned a year ago in the comments are still running, but I do make sure they are never used more than 80%

Re: Running out of disk space in production

#133
post #46

Earlier quoted context omitted.

Please name and shame those filesystems so that we will all be forewarned.

Any Copy-on-Write filesystem can run into this. There's always some way around it, but it can be problematic if you only have one device, can't remember the steps to fix a full filesystem, and can't look up the steps because you can't launch a browser without it trying to make some files!

> and can't look up the steps because you can't launch a browser

My early days of computing got easier when I had a second computer to look up the issues of the first computer.

Re: Running out of disk space in production

#134

I appreciate the last line > Note: this was written fully by me, human.

My fav line from the post is one above it. There is something very blunt and funny about it. > It’s difficult to reason under pressure. Experience, that I didn’t have here, would have helped.

Not me : "Experience is what you get after you needed it."

Re: Running out of disk space in production

#135
post #131

Earlier quoted context omitted.

My choice has always been `shred`: $ sudo truncate --size 1G /emergency-space $ sudo shred /emergency-space I find it widely available, even in tiny distros.

Very cool. Reading the man page I see, by default, shred does 3 iterations. Adding --iterations=1 makes it random enough for this purpose and faster.

Terrific, thank you!

Re: Running out of disk space in production

#136
post #122

Earlier quoted context omitted.

Because adding swap file is instantaneous, removing one that is in use can take a longtime unless you reboot the OS so you can't just nuke it quickly.

Let's get crazy 1. swapoff 2. drop disk cache (1) 3. panik!?! hrm, seems ok

Actually there should be a 'sync' as step 2 :)

Re: Running out of disk space in production

#137
post #113
post #25

Earlier quoted context omitted.

Better fill those files with random bytes, to ensure the filesystem doesn’t apply some “I don’t actually have to store all-zero blocks” sparse-file optimization. To my knowledge no non-compressing file system currently does this, but who knows about the future.

A good way to do this is to create a swap file, both because then you can use it as a swap file until you need to delete it and because swap files are required to not be sparse.

I'm not at a machine, linux doesn't zero the swap "file" does it? if i set vm.swappiness = 0, will your "trick" work if i never hit memory pressure?

Re: Running out of disk space in production

#138
post #122

Earlier quoted context omitted.

Because adding swap file is instantaneous, removing one that is in use can take a longtime unless you reboot the OS so you can't just nuke it quickly.

Let's get crazy 1. swapoff 2. drop disk cache (1) 3. panik!?! hrm, seems ok

it has been 3 days since esseph has nuked prod

Re: Running out of disk space in production

#139

I remember a story of an Oracle Database customer who had production broken for days until an Oracle support escalation led to identifying the problem as mere "No disk space left".

Or NTP, if something is not working df -h and date are the first commands I input. It's always lupu... I mean NTP or disk space.

i had to ssh to our public nextcloud and run the occ empty-all-trashes-immediately --force --all-users -f command, due to its disk filling up (2.5TB!) and postgres crashing, to boot. empty-trashes command and a reboot and it came right back up.

Re: Running out of disk space in production

#140
post #113

Earlier quoted context omitted.

A good way to do this is to create a swap file, both because then you can use it as a swap file until you need to delete it and because swap files are required to not be sparse.

I'm not at a machine, linux doesn't zero the swap "file" does it? if i set vm.swappiness = 0, will your "trick" work if i never hit memory pressure?

If you create the file with 'mkswap --file' it allocates the blocks. Trying to use 'swapon' with an existing sparse file won't remove the holes for you but does notice them and then refuse to use it.
Post reply on HN