Live data from Hacker News

Running out of disk space in production

alt-romes.github.io

41–50 of 141 posts

Re: Running out of disk space in production

#41

I remember a story of an Oracle Database customer who had production broken for days until an Oracle support escalation led to identifying the problem as mere "No disk space left".

Or NTP, if something is not working df -h and date are the first commands I input.

It's always lupu... I mean NTP or disk space.

Re: Running out of disk space in production

#42

> I rushed to run du -sh on everything I could, as that’s as good as I could manage. I recently came across gdu (1) and have installed/used it on every machine since then. [1]: https://github.com/dundee/gdu

gdu is really nice but ncdu, though slower, is very useful and is usually available on distro repos.

Re: Running out of disk space in production

#43
post #36

If you run nginx anyway, why not serve static files from nginx? No need for temporary files, no extra disk space. The authorization can probably be done somehow in nginx as well.

Even if your authorization is so sophisticated that nginx cannot do it, a common pattern I’ve seen is to support a special HTTP response header for the reverse proxy to read directly from disk after your custom authorization code completes. This trick dates back to at least 2010. The nginx version of this seemed to be called X-Accel-Redirect from a quick search.

Re: Running out of disk space in production

#44
Putting limits on folders where information may be added (with partitions or project quotas) is a proactive way to avoid that something misbehaves and fills the whole disk. Filling that partition or quota may still cause some problems, depending on the applications writing there, but the impact may be lower and easier to fix than running out of space for everything.

Re: Running out of disk space in production

#45
post #36

If you run nginx anyway, why not serve static files from nginx? No need for temporary files, no extra disk space. The authorization can probably be done somehow in nginx as well.

Yeah it's a bit odd to use a Haskell server to serve a static file which nginx then needs to buffer. You'd do much much better just serving the file out of nginx. You could authenticate requests using the very simple auth_request module:

https://nginx.org/en/docs/http/ngx_http_auth_request_module....

Re: Running out of disk space in production

#46
post #19
post #2

A neat trick I was told is to always have ballast files on your systems. Just a few GiB of zeros that you can delete in cases like this. This won't fix the problem, but will buy you time and free space for stuff like lock files so you can get a working system.

Some filesystems can be unable to delete a file if full. Something to be a bit worried about.

Please name and shame those filesystems so that we will all be forewarned.

Re: Running out of disk space in production

#48
post #2

A neat trick I was told is to always have ballast files on your systems. Just a few GiB of zeros that you can delete in cases like this. This won't fix the problem, but will buy you time and free space for stuff like lock files so you can get a working system.

Sounds like something straight out of Dilbert

Re: Running out of disk space in production

#50
post #2

A neat trick I was told is to always have ballast files on your systems. Just a few GiB of zeros that you can delete in cases like this. This won't fix the problem, but will buy you time and free space for stuff like lock files so you can get a working system.

Interesting strategy, can't believe I've never heard of this one before.

Would it be more pragmatic to allocate a swap file instead? Something that provides a theoretical benefit in the short term vs a static reservation.

Post reply on HN