Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

All my servers have an 8 GB empty file on disk

brianschrader.com

Post reply on HN
Print view View original

411–420 of 715 posts

Re: All my servers have an 8 GB empty file on disk

#411

Post by john_moscow »

I happen to run a couple of small servers myself and here's a better version of this approach. Create a cron job that will run a simple self-testing script once every few hours. My self-test does this:

1. Checks that all domains can be accessed via HTTP and HTTPS. If not, DNS might have died.

2. Checks that a few known CMS-generated pages contain some phrases they should contain. If not, SQL might have died.

3. Checks that the HTTPS certificate has enough runway left. If not, certbot might have died.

4. Sends a basic email message from my domain to a gmail account. Receives it via IMAP and sends a reply. Then, verifies the reply. This catches a whole bunch of mail-related issues.

5. Checks the free RAM and disk space. Updates an internal "dashboard" page and sends me an email if they are off.

It only took a couple of hours to hack this together and I must say, I get a much better night time sleep ever since.

Re: All my servers have an 8 GB empty file on disk

#412

Post by dahdum »

bluedino wrote: post #400
not2b wrote: post #392

Earlier quoted context omitted.

No, companies need to be able to point to an official retention policy that says in writing that emails older then x months or years get deleted. Most do (including my employer), and it's because of legal discovery. But it feels like we're lobotomizing ourselves, as often the reason some odd thing was done was based on a long-deleted email discussion.

And the right way to do it is with an archiving service/appliance

Archiving is likely solving the wrong problem, for legal reasons they don’t want those old emails hanging around.

Re: All my servers have an 8 GB empty file on disk

#415

Post by chime »

john_moscow wrote: post #411

I happen to run a couple of small servers myself and here's a better version of this approach. Create a cron job that will run a simple self-testing script once every few hours. My self-test does this: 1. Checks that all domains can be accessed via HTTP and HTTPS. If not, DNS might have died. 2. Checks that a few known CMS-generated pages contain some phrases they should contain. If not, SQL might have died. 3. Check…

This would be an awesome apt package that I could configure with a few options and install on any server.

Re: All my servers have an 8 GB empty file on disk

#416

Post by bastawhiz »

keeperofdakeys wrote: post #242
geocrasher wrote: post #124

For everyone saying "This isn't a real solution!" I'd like to explain why I think you're wrong. 1) It's not intended to be a Real Solution(tm). It's intended to buy the admin some time to solve the Real Issue. 2) Having a failsafe on standby such as this will save an admin's butt when it's 2am and PagerDuty won't shut up, and you're just awake enough to apply a temp fix and work on it in the morning. 3) Because "FIX…

I find that either a server needs more space, or has files that can be deleted. For the former you just increase the disk space, since most things are VMs these days and increasing space is easy. For the latter you can usually delete enough files to get the service back up before you start the proper cleanup. If you really need some reserve space (physical server), I'd much rather store it in a vg (or zfs/btrfs subvo…

> Will you remember the file exists at 2am?

As someone who has been woken up at 2am for this exact issue, emphatically yes. I would much rather be back in bed than trying to Google the command to find large files on disk.

Re: All my servers have an 8 GB empty file on disk

#417

Post by pydave »

ABeeSea wrote: post #300
twistedpair wrote: post #261

Earlier quoted context omitted.

I worked at a large company during a migration from Lotus to Outlook. We were told we'd get our current Lost email storage + 100MiB as a new email quota limit under Outlook. I made a bunch of 100MiB files of `/dev/random` noise (so they don't compress, compressed size was part of the quota) and emailed them to myself before the migration, to get a few GiB of quota buffer. My co-workers were constantly having to delet…

Email quotas aren’t just a cost thing. It forces deletion of files/communications that aren’t relevant anymore. The last thing the legal department wants is some executive’s laptop with 10 years of undeleted email to make it’s way to discovery.

Unfortunately, those goals are rarely communicated and accepted by the people they're imposed on.

My first full-time job had an unexplained email expiry policy. After being frustrated several times at losing some explanation on how/why, I started forwarding all my emails to gmail. In retrospect, that's probably a worse result to whoever imposed the expiration.

Fortunately, these days people are better about consolidating knowledge on wikis or some kind of shared docs instead of only email.

Re: All my servers have an 8 GB empty file on disk

#418

Post by nitrogen »

Cerium wrote: post #76
SethTro wrote: post #31

Same idea as this game development legend https://www.dodgycoder.net/2012/02/coding-tricks-of-game-dev... > he had put aside those two megabytes of memory early in the development cycle. He knew from experience that it was always impossible to cut content down to memory budgets, and that many projects had come close to failing because of it. So now, as a regular practice, he always put aside a nice block of memory to…

In my work it is very common to make the memory map a little smaller than it has to be. If you can't ship an initial version in a reduced footprint you will have no hope of shipping future bugfixes.

Many years ago I spent a couple of weeks fixing a firmware bug. The firmware was only a few dozen bytes shy of the EEPROM. I just #ifdef'd out a bunch of features to focus on debugging what was broken, but to get the fix released I had to manually optimize several other parts of the code to get everything to fit in the 2MB or whatever it was.

Would've been nice if someone had reserved some space ahead of time. Maybe they did, but nobody was around who remembered that codebase.

Re: All my servers have an 8 GB empty file on disk

#419

Post by Cullinet »

dasudasu wrote: post #367
tinus_hn wrote: post #114

Earlier quoted context omitted.

Same thing happens with floating licenses, if they are too scarce, people open the program first thing in the morning ‘just in case’ and keep a license reserved all day.

The real game starts when people run infinite while/for loops that try to check out one as soon as it's available. Or run useless operations within the licensed software just so that that the license doesn't expire and return to the pool. I'm guilty of both, sadly. In an academic environment, additional resources aren't going to fall from the sky.

ouch I can't stop thinking now about how much cost gets imposed on the economy by habits like this established in higher education - I built my original business with very few formally qualified people who included a large proportion of the most experienced and professionally qualified individuals including several with multinational boardroom careers in F500s. we didn't have the culture to tolerate games like holding up a floating licence (of which licence a lot of critical software used) and we weren't the generation raised with computers by a few distant, but hearing this both makes perfect sense that it might be prevalent and simultaneously is thoroughly unnerving me about how strongly I might react on encountering the same if my present venture gets going.
Post reply on HN