Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Running out of disk space in production

alt-romes.github.io

Post reply on HN
Print view View original

121–130 of 141 posts

Re: Running out of disk space in production

#121

Post by reddalo »

Dylan16807 wrote: post #91
reddalo wrote: post #90

Earlier quoted context omitted.

This trick is actually used by some banking apps. They fill app their mobile apps with junk data just to make the APK/IPA bigger. So if they need to push an urgent update, they won't have users that can't update because their phones are full to the brim. I know two Italian banks that do it, Unicredit and Intesa. The latter was on the news when a user found out that one of the filler files was a burp recording [1]. [1…

But you still need a bunch of extra space to download and unpack the new version, and there are so many apps that need to share space, and a banking app should only need about 0.1% of a phone's storage... Whoever gave them that idea was doing a bad deed.

I know and I agree with you. It doesn't seem that smart.

And you can tell by the fact that the filler data is called "burp.mp3" and things like that.

Re: Running out of disk space in production

#122

Post by esseph »

prmoustache wrote: post #65
happycrappy wrote: post #50

Earlier quoted context omitted.

Interesting strategy, can't believe I've never heard of this one before. Would it be more pragmatic to allocate a swap file instead? Something that provides a theoretical benefit in the short term vs a static reservation.

Because adding swap file is instantaneous, removing one that is in use can take a longtime unless you reboot the OS so you can't just nuke it quickly.

Let's get crazy

1. swapoff

2. drop disk cache (1)

3. panik!?!

hrm, seems ok

Re: Running out of disk space in production

#123

Post by Dylan16807 »

justsomehnguy wrote: post #115
Dylan16807 wrote: post #95

Earlier quoted context omitted.

Empty space is good for wear-leveling but enforcing a few percent extra helps. > And most importantly 10% of the drive in ~2010 were 6-12GB, nowadays it's 50-100GB at least. Back then you were paying about $2 per gigabyte. Right now SSDs are 1/15th as expensive. If we use the prices from last year they're 1/30th, and if we also factor in inflation it's around 1/50th. So while I would say to use a lower percentage as…

> but enforcing a few percent extra helps. Only if you fill the drive up to 95-99% and do this often. Otherwise it's just a cargo-cult. > So while I would say to use a lower percentage as space increases If your drive is over-provisioned (eg 960GB instead of 1024GB) then it's not needed. If not and you fill your drive to the full and just want to be sure then you need the size of the biggest write you would do plus s…

My drive gets almost full relatively often.

> If your drive is over-provisioned (eg 960GB instead of 1024GB) then it's not needed.

I disagree. That much space isn't a ton when it comes to absorbing the wear of background writes. And normal use ends up with garbage sectors sprinkled around inflating your data size, which makes write amplification get really bad as you approach 100% utilization and have to GC more and more. 6% extra is in the range where more will meaningfully help.

> Leaving 100GB of 1TB drive is like buying a sneakers but not wearing them because they would wear.

50GB is like $4 of space the last time most people bought an SSD. Babying the drive with $4 is very far from refusing to use it at all. The same for 100GB on a 4TB drive.

Re: Running out of disk space in production

#124

Post by PunchyHamster »

flanfly wrote: post #2

A neat trick I was told is to always have ballast files on your systems. Just a few GiB of zeros that you can delete in cases like this. This won't fix the problem, but will buy you time and free space for stuff like lock files so you can get a working system.

Just use LVM and don't allocate all of it to LV.

We have a script that basically slowly expands volume when demand grows, up to a limit. So we don't have to think on stuff like "does the logs partition need to be 1 or 10GB", it will expand to the sane limit, and if it hits that we get disk usage alert before it finishes so we can either see what's going on (app shat in logs), or take a look for the one in the 10 apps that need some special tuning there

Re: Running out of disk space in production

#125

Post by PunchyHamster »

layer8 wrote: post #25
flanfly wrote: post #2

A neat trick I was told is to always have ballast files on your systems. Just a few GiB of zeros that you can delete in cases like this. This won't fix the problem, but will buy you time and free space for stuff like lock files so you can get a working system.

Better fill those files with random bytes, to ensure the filesystem doesn’t apply some “I don’t actually have to store all-zero blocks” sparse-file optimization. To my knowledge no non-compressing file system currently does this, but who knows about the future.

No, just use LVM or other dynamic volume management.

Shit like that just wastes space that SSD could use for wear levelling...

Re: Running out of disk space in production

#126

Post by PunchyHamster »

dspillett wrote: post #15
flanfly wrote: post #2

A neat trick I was told is to always have ballast files on your systems. Just a few GiB of zeros that you can delete in cases like this. This won't fix the problem, but will buy you time and free space for stuff like lock files so you can get a working system.

Similarly, I always leave some space unallocated on LMV volume groups. It means that I can temporarily expand a volume easily if needed. It also serves to leave some space unused to help out the wear-levelling on the SSDs on which the RAID array that is the PV¹ for LVM. I'm, not 100% sure this is needed any more² but I've not looked into that sufficiently so until I do I'll keep the habit. -------- [1] if there are m…

I do that + script to auto resize within sane limits. So for most servers the partitions will automatically fit the usage while still leaving some spare space.

Usually something like "expand if there is less than 5% left, with monitoring triggering when there is 4% free space left", so there is still warning when the automatic resize is on limit

carving space per PV like that is pointless

> It also serves to leave some space unused to help out the wear-levelling on the SSDs on which the RAID array that is the PV¹ for LVM. I'm, not 100% sure this is needed any more² but I've not looked into that sufficiently so until I do I'll keep the habit.

YMMV but most distros set up a cron/timer that does fstrim monthly. So it shouldn't be needed, as any free space will be returned to SSD.

> [1] if there are multiple PVs, from different drives/arrays, in the VG, then you might need to manually skip a bit on each one because LVM will naturally fill one before using the next. Just allocate a small LV specially on each and don't use it. You can remove one/all of them and add the extents to the fill LV if/when needed. Giving it a useful name also reminds you why that bit of space is carved out.

other options is telling LVM this LV is striped (so it uses space from both drives equally), or manually allocating from drive with more free space when expanding/adding LV

Re: Running out of disk space in production

#127

Post by PunchyHamster »

justsomehnguy wrote: post #115
Dylan16807 wrote: post #95

Earlier quoted context omitted.

Empty space is good for wear-leveling but enforcing a few percent extra helps. > And most importantly 10% of the drive in ~2010 were 6-12GB, nowadays it's 50-100GB at least. Back then you were paying about $2 per gigabyte. Right now SSDs are 1/15th as expensive. If we use the prices from last year they're 1/30th, and if we also factor in inflation it's around 1/50th. So while I would say to use a lower percentage as…

> but enforcing a few percent extra helps. Only if you fill the drive up to 95-99% and do this often. Otherwise it's just a cargo-cult. > So while I would say to use a lower percentage as space increases If your drive is over-provisioned (eg 960GB instead of 1024GB) then it's not needed. If not and you fill your drive to the full and just want to be sure then you need the size of the biggest write you would do plus s…

Nah, we used some consumer SSD for write heavy but not all that precious data, and time to live was basically directly dependant on the space left free on device.

Of course, doesn't matter for desktop use as the spare on drive is enough, but still, if you have 24/7 write heavy loads, making sure it's all trimmed will noticably extend lifetime

Re: Running out of disk space in production

#128

Post by ape4 »

bugfix wrote: post #86
ape4 wrote: post #30

Earlier quoted context omitted.

If I recall correctly: dd if=/dev/urandom of=/home/myrandomfile bs=1 count=N

I just use fallocate to create a 1GB or 2GB file, depending on the total storage size. It has saved me twice now. I had a nasty issue with a docker container log quickly filling up the 1GB space before I could even identify the problem, causing the shell to break down and commands to fail. After that, I started creating a 2GB file.

Interesting! I did `fallocate -l 1G myfile` - very fast. Its all zeros but probably won't be compressed by the filesystem since its created with the fallocate() system call.

Re: Running out of disk space in production

#130

Post by functional_dev »

flanfly wrote: post #2

A neat trick I was told is to always have ballast files on your systems. Just a few GiB of zeros that you can delete in cases like this. This won't fix the problem, but will buy you time and free space for stuff like lock files so you can get a working system.

Such a simple trick, but effective!
Post reply on HN