This is stupid. Just don't make your servers use up all space. That's why ext can reserve space for root, 5% by default.
All my servers have an 8 GB empty file on disk
341–350 of 715 posts
Re: All my servers have an 8 GB empty file on disk
#342One thing that many Linux/Unix users do not know is that all commonly used filesystems have a "reserved" amount of space to which only "root" can write. The typical format (mkfs) default is to leave 5% of the disk reserved. The reserved space can be modified (by root) any time, and it can be specified as a block count or a percentage. As long as your application does not have root privileges, it will hit the wall whe…
Of course the application is running as root, duh.
Re: All my servers have an 8 GB empty file on disk
#343Earlier quoted context omitted.
> Property ownership comes at a cost. Sort-of. Typically there is a cost associated with getting rights over a property, either by manufacturing, or in the case of land, by purchase or through the efforts of settling. However, once you have ownership of a property, it's usually relatively cheap to continue own (except, I suppose, if you consider the risk of a communist revolution or something). This is esentially how…
In California you pay ~1.5% in property taxes per year. Owning property is not cheap. In fact you never really own it.
Unless you're a golf course in LA, and cajole a permanent exemption!
Re: All my servers have an 8 GB empty file on disk
#344> On Linux servers it can be incredibly difficult for any process to succeed if the disk is full. Copy commands and even deletions can fail or take forever as memory tries to swap to a full disk and there's very little you can do to free up large chunks of space. This reasoning doesn't make sense. On Linux, swap is preallocated. This is true regardless of whether you're using a swap partition or a swap file. See man…
Not sure about their reasoning.. but if you don't have root ssh enabled, sudo can break if there is no free disk space. I do something similar where I write a 500mb file to /tmp and chmod 777 it so anyone can free it up without needing sudo.
In that case, use "su".
Re: All my servers have an 8 GB empty file on disk
#345Earlier quoted context omitted.
Yes because you could just do > save_my_butt.img and now it's 0 bytes.
Would that work? The fs may actually allocate a new file before deleting the exisiting allocation so the risk of it not working is still there I would think?
Worst case scenario, if
> filename
didn't null it, then just
echo "0" > filename
Re: All my servers have an 8 GB empty file on disk
#346> On Linux servers it can be incredibly difficult for any process to succeed if the disk is full. Copy commands and even deletions can fail or take forever as memory tries to swap to a full disk and there's very little you can do to free up large chunks of space. This reasoning doesn't make sense. On Linux, swap is preallocated. This is true regardless of whether you're using a swap partition or a swap file. See man…
IIRC, swap is actually needed for some memory operations. And when you run out of memory, the behaviour is often worse without swap. These days I always at least configure an in-memory compressed swap (zram).
You recall incorrectly; swap is not needed. It's not just me who runs without it; Google production machines did for many years.
"The behavior is often worse without swap" is more vague / subjective. I prefer that a process die cleanly than everything slow to a crawl semi-permanently. I've previously written about swap causing the latter: https://news.ycombinator.com/item?id=13715917 To some extent the bad behavior can happen even without swap because non-locked file-backed pages get paged in slowly also, but it seems significantly worse with swap.
zram is a decent idea though. I use it on my memory-limited Raspberry Pi machines.
Re: All my servers have an 8 GB empty file on disk
#347Earlier quoted context omitted.
Email quotas aren’t just a cost thing. It forces deletion of files/communications that aren’t relevant anymore. The last thing the legal department wants is some executive’s laptop with 10 years of undeleted email to make it’s way to discovery.
Then why not just tell Exchange to delete any emails older than 5 years (or whatever your lawyers tell you to put)?
A previous company I worked for had a one month retention window in the email server. People just ended up storing email in their local machine's Outlook folder so they can refer to old emails.
Re: All my servers have an 8 GB empty file on disk
#348Earlier quoted context omitted.
I still know people that pre-emptively buy toilet paper "because hoarders might buy the rest of it" with absolutely no introspection.
The trick is to do exactly what they tell you not too immediately. Masks don't protect against covid - jump on Alibaba and buy some. Don't buy extra toiled paper, there will be no limits on its sale - immediately do a order on amazon prime. Because within a few days there will be price caps on the sale of TP (increased prices would deter hoarding) so everything will sell out right away. I think people have really lea…
Interesting to think about this in the context of inflation - "We do expect some price increases as the economy warms up but that's not inflation."
Re: All my servers have an 8 GB empty file on disk
#349Earlier quoted context omitted.
I still know people that pre-emptively buy toilet paper "because hoarders might buy the rest of it" with absolutely no introspection.
The trick is to do exactly what they tell you not too immediately. Masks don't protect against covid - jump on Alibaba and buy some. Don't buy extra toiled paper, there will be no limits on its sale - immediately do a order on amazon prime. Because within a few days there will be price caps on the sale of TP (increased prices would deter hoarding) so everything will sell out right away. I think people have really lea…
I think you’ve just described a self-fulfilling prophecy. TP is running out because enough people think like you, turning a not-scarce commodity into a scarce commodity.
Re: All my servers have an 8 GB empty file on disk
#350A lot of tips in this thread are about how to better alert when you get low on disk space, how to recover, etc. but I'd like to highlight the statement: "The disk filled up, and that's one thing you don't want on a Linux server—or a Mac for that matter. When the disk is full nothing good happens." As developers, we need to be better at handling edge cases like out of disk space, out of memory, pegged bandwidth and pe…
A lot of measures are preventative, and kind of have to be.
Consider the hypothetical scenario of being totally out of memory. I mean completely: not a single page free, all buffers and caches flushed, everything else taken up by data that cannot be evicted. So in result, you cannot spawn a process. You cannot do any filesystem operations that would end up in allocations. You can't even get new page tables.
Hence things like Linux's OOM killer, which judiciously kills processes--not necessarily the ones you would like killed in such a situation. And again, a lot of preventative measures to not let it come that far.
Our Turing Machines still want infinite tapes, in a way.