Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

All my servers have an 8 GB empty file on disk

brianschrader.com

Post reply on HN
Print view View original

191–200 of 715 posts

Re: All my servers have an 8 GB empty file on disk

#191

Post by comex »

kiwijamo wrote: post #122
derefr wrote: post #72

Earlier quoted context omitted.

On systemd systems, logfiles are written to disk under the journald user, `systemd-journal`.

Is that true for all logfiles? I still have plenty of daemons (by default) writing directly to some file in /var/log eg EXIM, Apache, and the like. Also plenty of system stuff still write to files in that directory. And yes this is a machine that uses systemd.

But those daemons don’t usually have their own log writer processes running as root, do they? Instead, either the log file is accessible by the user the daemon is running as, or the daemon opens the log file as root before dropping privileges for the rest of its operation.

Re: All my servers have an 8 GB empty file on disk

#192

Post by stonesweep »

kiwijamo wrote: post #122
derefr wrote: post #72

Earlier quoted context omitted.

On systemd systems, logfiles are written to disk under the journald user, `systemd-journal`.

Is that true for all logfiles? I still have plenty of daemons (by default) writing directly to some file in /var/log eg EXIM, Apache, and the like. Also plenty of system stuff still write to files in that directory. And yes this is a machine that uses systemd.

Most vendors (Debian/Ubuntu, RHEL/clones, etc.) add a hook into rsyslog to be a partner with the systemd logger and write out text files next to the journal - they realize that a lot of people dislike dealing with journalctl (I'm one of them) and provide an alternate hook already installed and working for you behind the scenes.

This is for daemons using syslog methodology, not direct writers like apache/nginx/mysql/etc; think more like cron, systemd, chrony, NetworkManager, and so forth. The vendors are not all aligned on what goes where (example: on RHEL, pacemaker/crm write to their own logs buy on openSUSE they're sent to syslog) - the actual results differ slightly from vendor to vendor.

DIY distros like Arch do not implement the rsyslog backend by default, you have to set it up yourself following the wiki - only journalctl is there by default.

Re: All my servers have an 8 GB empty file on disk

#193

Post by dnautics »

0_____0 wrote: post #169
tehjoker wrote: post #141

Earlier quoted context omitted.

This reminds me of the reserve tank toggle on some motorcycles. When you run out of gas, you switch the toggle and drive directly to a gas station.

always fun when you're barreling down the highway and the engine starts to lean out, prompting you to hurriedly locate and switch the petcock over before the engine stalls completely. suppose then that you go fill up and forget to set the petcock back to normal. 8ball says: "I see a long walk in your future."

Surprised there isn't a mechanism that mechanically switches the petcock over when you put a fuel nozzle up to the port

Re: All my servers have an 8 GB empty file on disk

#195

Post by dmingod666 »

rozap wrote: post #118
diego wrote: post #54

This points to a much more serious problem. This is 2021 and the technology is from the 90s, with a really poor user experience design. Your car warns you when you're low on fuel, but your server doesn't if you're low on critical resources.

Everyone has this kind of alerting set up, but that's not the point. The beauty of this solution is that it's dead simple and will never fail. Alerting can fail or be ignored. It's the same as old VW beetles which had a reserve gas tank. When you ran out of gas you opened a valve and you could limp to a gas station. Less likely to fail versus a 1950's era gauge that is telling you you're low. Also impossible to ignor…

The 'beauty' artificially chokes your HDD and produces the same problems that you are trying to avoid.. not a sane way to proactively manage your disk usage.

Re: All my servers have an 8 GB empty file on disk

#196

Post by ineedasername »

dmingod666 wrote: post #184
diego wrote: post #54

This points to a much more serious problem. This is 2021 and the technology is from the 90s, with a really poor user experience design. Your car warns you when you're low on fuel, but your server doesn't if you're low on critical resources.

Exactly, it's 1990s 'cool' - the time it took him to write the blog, he could have written a script that would send him updates on all his devices...

There's no reason not to have multiple fail-safes. Receiving the alert on a device at 3am would still mean he could free up 8gb immediately and have breathing room to solve the problem. And remember this is for a single admin. Asking such a person to be on call 24-7 all year, vacations, holidays, weekends... Having a quick way to get breathing room can significantly reduce the stress & cognitive load of worrying about such things in your off-time.

Re: All my servers have an 8 GB empty file on disk

#197

Post by kiwijamo »

geocrasher wrote: post #107
capableweb wrote: post #83

I've had my disks so full that a `rm` command doesn't even work, would this workaround work in those cases too?

Yes because you could just do > save_my_butt.img and now it's 0 bytes.

Would that work? The fs may actually allocate a new file before deleting the exisiting allocation so the risk of it not working is still there I would think?

Re: All my servers have an 8 GB empty file on disk

#198

Post by qwertox »

outworlder wrote: post #50
fractal618 wrote: post #43

I think i know where this is going without even reading. Any attempt from the outside to pull this 8b Gb file would be a very noticeable red flag.

Nope. Try again :)

But the idea isn't that bad. Name it properly, like saudi-arabia-customer-data.sql.pgp next to a directory named pgp-keys and fill it with /dev/random.

Re: All my servers have an 8 GB empty file on disk

#199

Post by dylan604 »

kernelsanderz wrote: post #49
1123581321 wrote: post #6

This is clever. Our shaky version of this, historically, has been to run ncdu and hastily delete the first large log file we see. It’s not ideal.

ncdu saves my bacon at least every few months. I do machine learning and am always running out of space!

sounds like your machine doesn't seem to be learning the right things.

Re: All my servers have an 8 GB empty file on disk

#200

Post by tetha »

pritambarhate wrote: post #89

All my servers have an alarm when disk space goes above 70%. It sends an email every hour once the disk usage goes above 70%. Never had a server go down because of disk space issue after adopting this practise. Also one of the main reasons server disks go full is generally log files. Always remember to "logrotate" your log files and you will not have this issue that much. Yes one more thing, for all user uploaded fil…

This goes into the same vein I was going to point out.

Most uncontrolled space usage comes from logs, users doing user things, or something like build servers just eating temporary and caching storage for lunch. Databases also tend to have uncontrolled space usage, but that tends to be wanted.

So, if you push /var/log to it's own 20-30Gb partition, a mad logger cannot fill up /. It can kill logging, but no logging is better than fighting with a full /. Similar things with /home - let users fill up their home dirs and pout and scream about it... but / is still fine. And you can use their input to provide more storage, if they have useful workflows.

Something like databases - where their primary use case is to grow - need monitoring though to add storage as necessary.

Post reply on HN