Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Running out of disk space in production

alt-romes.github.io

Post reply on HN
Print view View original

91–100 of 141 posts

Re: Running out of disk space in production

#91

Post by Dylan16807 »

reddalo wrote: post #90
flanfly wrote: post #2

A neat trick I was told is to always have ballast files on your systems. Just a few GiB of zeros that you can delete in cases like this. This won't fix the problem, but will buy you time and free space for stuff like lock files so you can get a working system.

This trick is actually used by some banking apps. They fill app their mobile apps with junk data just to make the APK/IPA bigger. So if they need to push an urgent update, they won't have users that can't update because their phones are full to the brim. I know two Italian banks that do it, Unicredit and Intesa. The latter was on the news when a user found out that one of the filler files was a burp recording [1]. [1…

But you still need a bunch of extra space to download and unpack the new version, and there are so many apps that need to share space, and a banking app should only need about 0.1% of a phone's storage...

Whoever gave them that idea was doing a bad deed.

Re: Running out of disk space in production

#93

Post by seized »

bombcar wrote: post #19
flanfly wrote: post #2

A neat trick I was told is to always have ballast files on your systems. Just a few GiB of zeros that you can delete in cases like this. This won't fix the problem, but will buy you time and free space for stuff like lock files so you can get a working system.

Some filesystems can be unable to delete a file if full. Something to be a bit worried about.

Instead of deleting the ballast file you can just truncate it. That works on ZFS when you fill the pool and delete starts failing.

Re: Running out of disk space in production

#94

Post by ikr678 »

reddalo wrote: post #90
flanfly wrote: post #2

A neat trick I was told is to always have ballast files on your systems. Just a few GiB of zeros that you can delete in cases like this. This won't fix the problem, but will buy you time and free space for stuff like lock files so you can get a working system.

This trick is actually used by some banking apps. They fill app their mobile apps with junk data just to make the APK/IPA bigger. So if they need to push an urgent update, they won't have users that can't update because their phones are full to the brim. I know two Italian banks that do it, Unicredit and Intesa. The latter was on the news when a user found out that one of the filler files was a burp recording [1]. [1…

Doesnt this create an arms race situation where every 'critical' app claims a larger diskspace than necessary, just in case, and accelerates the issue?

Re: Running out of disk space in production

#95

Post by Dylan16807 »

justsomehnguy wrote: post #61
dspillett wrote: post #15

Earlier quoted context omitted.

Similarly, I always leave some space unallocated on LMV volume groups. It means that I can temporarily expand a volume easily if needed. It also serves to leave some space unused to help out the wear-levelling on the SSDs on which the RAID array that is the PV¹ for LVM. I'm, not 100% sure this is needed any more² but I've not looked into that sufficiently so until I do I'll keep the habit. -------- [1] if there are m…

Not needed. All your unused/unfilled space is that space for wear-leveling. It wasn't needed even back then besides some corner cases. And most importantly 10% of the drive in ~2010 were 6-12GB, nowadays it's 50-100GB at least.

Empty space is good for wear-leveling but enforcing a few percent extra helps.

> And most importantly 10% of the drive in ~2010 were 6-12GB, nowadays it's 50-100GB at least.

Back then you were paying about $2 per gigabyte. Right now SSDs are 1/15th as expensive. If we use the prices from last year they're 1/30th, and if we also factor in inflation it's around 1/50th.

So while I would say to use a lower percentage as space increases, 50-100GB is no problem at all.

Re: Running out of disk space in production

#96

Post by majormajor »

theshrike79 wrote: post #9
omarqureshi wrote: post #4

Earlier quoted context omitted.

Surely a 50% warning alarm on disk usage covers this without manual intervention?

Depends. A Kubernetes container might have only a few megabytes of disk space, because it shouldn't need it. Except that one time when .NET decides that the incoming POST is over some magic limit and it doesn't do the processing in-memory like before, but instead has to write it to disk, crashing the whole pod. Fun times. Also my Unraid NAS has two drives in "WARNING! 98% USED" alert state. One has 200GB of free spac…

The "ballast file" idea doesn't really change that spill-to-disk crash, as far as I can tell. You have to delete it manually; it already crashed by the time you realize it.

Seems like the sort of thing that only makes sense in a "I know my cheapskate boss won't have larger drives ready to go (or be willing to pay to expand it in a cloud scenario), and he insists that the alarm not go off until 95%, but it'll be my fault if we have a bad incident we can't recover quickly from, so I'm gonna give myself some headroom by padding things a bit" extra-paranoid scenario.

Re: Running out of disk space in production

#97

Post by majormajor »

evil-olive wrote: post #60
omarqureshi wrote: post #4

Earlier quoted context omitted.

Surely a 50% warning alarm on disk usage covers this without manual intervention?

> Surely a 50% warning alarm on disk usage covers this without manual intervention? surely you don't need a fire extinguisher in your kitchen, if you have a smoke detector? a "warning alarm" is a terrible concept, in general. it's a perfect way to lead to alert fatigue. over time, you're likely to have someone silence the alarm because there's some host sitting at 57% disk usage for totally normal reasons and they're…

How does the ballast file prevent extreme runaway? You ain't gonna notice and delete it that quickly.

Re: Running out of disk space in production

#98

Post by veverkap »

ilaksh wrote: post #59

I'm not sure that his problems are really over if a LOT of people were downloading a 2GB file. It would depend on the plan. Especially if his server is in the US. But maybe the European Hetzner servers still have really big limits even for small ones. But still, if people keep downloading, that could add up.

I was thinking the same thing - wouldn't blob storage or a CDN help?

Re: Running out of disk space in production

#99

Post by bentcorner »

ikr678 wrote: post #94
reddalo wrote: post #90

Earlier quoted context omitted.

This trick is actually used by some banking apps. They fill app their mobile apps with junk data just to make the APK/IPA bigger. So if they need to push an urgent update, they won't have users that can't update because their phones are full to the brim. I know two Italian banks that do it, Unicredit and Intesa. The latter was on the news when a user found out that one of the filler files was a burp recording [1]. [1…

Doesnt this create an arms race situation where every 'critical' app claims a larger diskspace than necessary, just in case, and accelerates the issue?

Kinda sorta, but there's a limit where users will typically install X apps and apps of Y size need Z extra space to update. User content would fill up the rest. I would imagine a typical 256 gb phone is probably over this limit and people who take lots of videos/photos just need to clean up their phone a little more often.

Re: Running out of disk space in production

#100

Post by dotancohen »

ape4 wrote: post #30
layer8 wrote: post #25

Earlier quoted context omitted.

Better fill those files with random bytes, to ensure the filesystem doesn’t apply some “I don’t actually have to store all-zero blocks” sparse-file optimization. To my knowledge no non-compressing file system currently does this, but who knows about the future.

If I recall correctly: dd if=/dev/urandom of=/home/myrandomfile bs=1 count=N

My choice has always been `shred`:

  $ sudo truncate --size 1G /emergency-space
  $ sudo shred /emergency-space
I find it widely available, even in tiny distros.
Post reply on HN