Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

All my servers have an 8 GB empty file on disk

brianschrader.com

Post reply on HN
Print view View original

431–440 of 715 posts

Re: All my servers have an 8 GB empty file on disk

#431

Post by bostonsre »

luckylion wrote: post #216
bostonsre wrote: post #86

Earlier quoted context omitted.

Not all environments require a stringent SLA. I have some servers that don't have a stringent SLA and aren't worth being woken up at night over if their disk is filling up fast.

Okay, so you wake up with a full disk. What did the spacer accomplish?

It allows me to remove that big file, then I'm able to run sudo since I don't allow root ssh and sudo won't work with a full disk, then I can clear up space on the system, bring it up again, then update log rotate or do whatever to prevent that case from happening again.

Re: All my servers have an 8 GB empty file on disk

#432

Post by truetraveller »

tshaddox wrote: post #99

How is this better than sounding alarms when free disk space drops below 8GB? If you’re going to ignore the alarms, then you’re going to have the same problem after you remove your spacer file and the disk fills up again!

I believe you are correct. The other replies are really not addressing your critique at all. Both solutions require some form of "alerting", when not just do it the proper way. Worried about how long an alert will take to respsond to? Well, alert when 16GB remaining, you just bought yourself more time!

Re: All my servers have an 8 GB empty file on disk

#433

Post by o-__-o »

pydave wrote: post #417
ABeeSea wrote: post #300

Earlier quoted context omitted.

Email quotas aren’t just a cost thing. It forces deletion of files/communications that aren’t relevant anymore. The last thing the legal department wants is some executive’s laptop with 10 years of undeleted email to make it’s way to discovery.

Unfortunately, those goals are rarely communicated and accepted by the people they're imposed on. My first full-time job had an unexplained email expiry policy. After being frustrated several times at losing some explanation on how/why, I started forwarding all my emails to gmail. In retrospect, that's probably a worse result to whoever imposed the expiration. Fortunately, these days people are better about consolida…

It’s a hush hush kind of thing. You advertise it’s to avoid discovery and you are openly admiting to liability should someone find out while trying to pull your execs email during discovery.

The excuse of resource contention provides plausible deniability

Re: All my servers have an 8 GB empty file on disk

#434

Post by truetraveller »

This is not the right solution. It's like setting your clock 5 minutes ahead, to trick yourself into thinking it's 9:00 am, when it's really 8:55 am. It doesn't work.

The better solution is simple monitoring. Alert when limit is passed. Increase limit to 16gb disk space remaining if paranoid.

Re: All my servers have an 8 GB empty file on disk

#435

Post by o-__-o »

phaer wrote: post #356
easton wrote: post #336

Earlier quoted context omitted.

Then why not just tell Exchange to delete any emails older than 5 years (or whatever your lawyers tell you to put)?

I knew a place where Exchange was configured to delete all mails after 6 months. Soon after I discovered that people started to form circles in which they would forward older mails from internal mailing lists to each other to retain them longer than that.

Fannie Mae did this. When you have targets on your back you minimize the collateral damage from possible blowback.

Imagine getting sued and having the entire paper trail in your email going back 3+ years. I expire all email after 1 year.

Re: All my servers have an 8 GB empty file on disk

#436

Post by bostonsre »

franga2000 wrote: post #295
bostonsre wrote: post #78

Earlier quoted context omitted.

Not sure about their reasoning.. but if you don't have root ssh enabled, sudo can break if there is no free disk space. I do something similar where I write a 500mb file to /tmp and chmod 777 it so anyone can free it up without needing sudo.

I've experienced far more full disks than I'd want to admit, on many different hardware and software configurations, and I've never seen sudo break. Is this something you've experienced recently? I definitely agree with your advice and will go double check all my servers if /filler is 777 (not in /tmp since it's sometimes mounted tmpfs), but if sudo does break in that situation, that sounds like a pretty severe and m…

Yea, have seen it recently on centos 7. I think it's due to having sudo logging enabled. It won't let you run sudo without it working (at least I think that's the case, but haven't spent the time to investigate too thoroughly).

Re: All my servers have an 8 GB empty file on disk

#437

Post by KernelMC »

I keep my databases on a separate file system from root, var, or anything system critical for this reason. Wouldn't you still have down time when you fill up the filesystem with the 8GB space waster in place? I might be missing something here, but this doesn't seem like a good solution.

Re: All my servers have an 8 GB empty file on disk

#439

Post by Cacti »

danielrhodes wrote: post #80

My understanding is this is why one should partition a drive. If you have a data partition, a swap partition, and an OS partition, you can get around issues where a server’s lack of disk space hoses the whole system.

Do people not create partitions any more? I thought this was sysadmin 101 for, like, forever. Databases, web servers, etc. should never ever fill up the entire disk. Separate partitions for boot, swap, /, /home, /var, and /tmp are the minimum common sense partitions.

Re: All my servers have an 8 GB empty file on disk

#440

Post by bostonsre »

reph2097 wrote: post #344
bostonsre wrote: post #78

Earlier quoted context omitted.

Not sure about their reasoning.. but if you don't have root ssh enabled, sudo can break if there is no free disk space. I do something similar where I write a 500mb file to /tmp and chmod 777 it so anyone can free it up without needing sudo.

In that case, use "su".

No luck when I've tried that. I think it's tied to sudo logging not working when the disk is full.
Post reply on HN