Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Running out of disk space in production

alt-romes.github.io

Post reply on HN
Print view View original

101–110 of 141 posts

Re: Running out of disk space in production

#101

Post by dotancohen »

coredog64 wrote: post #38
omarqureshi wrote: post #4

Earlier quoted context omitted.

Surely a 50% warning alarm on disk usage covers this without manual intervention?

You don't want an alarm on a usage threshold, you want a linear regression that predicts when utilization will cross a threshold. Then you set your alarms for "How long does it take me to remediate this condition?"

That's far more complicated and fragile. Where are you storing this log of disk usage? If you already have some external time series database then this is already a solved problem. But for a single server, desktop, or embedded device you'll need a database or text log, a cron job to measure it, and another script to parse, make predictions, and then raise alerts.

And a single large dump to disk, like some daemon suddenly bugging out and writing incessantly to logs, will render all that moot anyway.

Re: Running out of disk space in production

#102

Post by watermelon0 »

veverkap wrote: post #98
ilaksh wrote: post #59

I'm not sure that his problems are really over if a LOT of people were downloading a 2GB file. It would depend on the plan. Especially if his server is in the US. But maybe the European Hetzner servers still have really big limits even for small ones. But still, if people keep downloading, that could add up.

I was thinking the same thing - wouldn't blob storage or a CDN help?

European Hetzner VPSes have at least 20 TB of bandwidth, and US ones have at least 1 TB.

I don't think there is a cheaper CDN.

Re: Running out of disk space in production

#103

Post by dabinat »

It can be difficult to reason about seemingly innocuous things at scale. I have definitely fallen into the trap of increasing file size from 8 KB to 10 KB and having it cause massive problems when multiplied across all customers at once.

Re: Running out of disk space in production

#104

Post by snthpy »

flanfly wrote: post #2

A neat trick I was told is to always have ballast files on your systems. Just a few GiB of zeros that you can delete in cases like this. This won't fix the problem, but will buy you time and free space for stuff like lock files so you can get a working system.

Great idea, thanks!

Re: Running out of disk space in production

#105

Post by tempestn »

ssl-3 wrote: post #88
evil-olive wrote: post #60

Earlier quoted context omitted.

> Surely a 50% warning alarm on disk usage covers this without manual intervention? surely you don't need a fire extinguisher in your kitchen, if you have a smoke detector? a "warning alarm" is a terrible concept, in general. it's a perfect way to lead to alert fatigue. over time, you're likely to have someone silence the alarm because there's some host sitting at 57% disk usage for totally normal reasons and they're…

50% is probably unrealistic. Nobody really wants to diminish their storage by 50%. Let's set a fixed threshold -- 100GB, say -- and play out both methods. Method A: One or more ballast files are created, totalling 100GB. The machine runs out of storage and grinds to a halt. Hopefully someone notices soon or gets a generic alert that it has ceased, remembers that there's ballast files, and deletes one or more of them.…

A + B would be best. Warn at 200, file to reserve the last 100 (or 50 or whatever). That way if the fill is too fast to react to in time, you still have a quick way to temporarily gain disk space, if needed to solve the problem.

Re: Running out of disk space in production

#106

Post by tempestn »

majormajor wrote: post #97
evil-olive wrote: post #60

Earlier quoted context omitted.

> Surely a 50% warning alarm on disk usage covers this without manual intervention? surely you don't need a fire extinguisher in your kitchen, if you have a smoke detector? a "warning alarm" is a terrible concept, in general. it's a perfect way to lead to alert fatigue. over time, you're likely to have someone silence the alarm because there's some host sitting at 57% disk usage for totally normal reasons and they're…

How does the ballast file prevent extreme runaway? You ain't gonna notice and delete it that quickly.

It doesn't prevent it. It gives you a way to potentially recover after the disk fills. Many operations become impossible once the disk is full, so this buys you some temporary breathing room to solve the problem.

Re: Running out of disk space in production

#107

Post by ssl-3 »

tempestn wrote: post #105
ssl-3 wrote: post #88

Earlier quoted context omitted.

50% is probably unrealistic. Nobody really wants to diminish their storage by 50%. Let's set a fixed threshold -- 100GB, say -- and play out both methods. Method A: One or more ballast files are created, totalling 100GB. The machine runs out of storage and grinds to a halt. Hopefully someone notices soon or gets a generic alert that it has ceased, remembers that there's ballast files, and deletes one or more of them.…

A + B would be best. Warn at 200, file to reserve the last 100 (or 50 or whatever). That way if the fill is too fast to react to in time, you still have a quick way to temporarily gain disk space, if needed to solve the problem.

I like that idea. Belt and suspenders.

Alerting on an unexpectedly high rate-of-change, as some others have suggested, also seems good for some workloads.

Re: Running out of disk space in production

#108

Post by johntash »

flanfly wrote: post #2

A neat trick I was told is to always have ballast files on your systems. Just a few GiB of zeros that you can delete in cases like this. This won't fix the problem, but will buy you time and free space for stuff like lock files so you can get a working system.

ext2/ext3/ext4 all automatically reserve an amount of space on a partition. 5% iirc

Re: Running out of disk space in production

#109

Post by triage8004 »

reddalo wrote: post #90
flanfly wrote: post #2

A neat trick I was told is to always have ballast files on your systems. Just a few GiB of zeros that you can delete in cases like this. This won't fix the problem, but will buy you time and free space for stuff like lock files so you can get a working system.

This trick is actually used by some banking apps. They fill app their mobile apps with junk data just to make the APK/IPA bigger. So if they need to push an urgent update, they won't have users that can't update because their phones are full to the brim. I know two Italian banks that do it, Unicredit and Intesa. The latter was on the news when a user found out that one of the filler files was a burp recording [1]. [1…

Interesting, makes sense, seems to be bad precedent if everyone follows suit.

Re: Running out of disk space in production

#110

Post by totetsu »

flanfly wrote: post #2

A neat trick I was told is to always have ballast files on your systems. Just a few GiB of zeros that you can delete in cases like this. This won't fix the problem, but will buy you time and free space for stuff like lock files so you can get a working system.

Also good for stopping phone-home auto firmware updates
Post reply on HN