Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

All my servers have an 8 GB empty file on disk

brianschrader.com

Post reply on HN
Print view View original

151–160 of 715 posts

Re: All my servers have an 8 GB empty file on disk

#151

Post by smarx007 »

luckylion wrote: post #131
geocrasher wrote: post #124

For everyone saying "This isn't a real solution!" I'd like to explain why I think you're wrong. 1) It's not intended to be a Real Solution(tm). It's intended to buy the admin some time to solve the Real Issue. 2) Having a failsafe on standby such as this will save an admin's butt when it's 2am and PagerDuty won't shut up, and you're just awake enough to apply a temp fix and work on it in the morning. 3) Because "FIX…

> 1) It's not intended to be a Real Solution(tm). It's intended to buy the admin some time to solve the Real Issue. If you don't have monitoring, will you even be aware that your disk is filling up? If you do have monitoring, why are you artificially filling up your disk so that it will be at 100% more quickly instead of just setting your monitoring up to alert you when it's at $whateverItWasSetToMinusEightGB?

An extra failsafe? You can do both. What if your cron/netdata are not forwarding emails for some reason (eg nullmailer gets errors from Mailgun)?

Re: All my servers have an 8 GB empty file on disk

#153

Post by dilyevsky »

geocrasher wrote: post #124

For everyone saying "This isn't a real solution!" I'd like to explain why I think you're wrong. 1) It's not intended to be a Real Solution(tm). It's intended to buy the admin some time to solve the Real Issue. 2) Having a failsafe on standby such as this will save an admin's butt when it's 2am and PagerDuty won't shut up, and you're just awake enough to apply a temp fix and work on it in the morning. 3) Because "FIX…

Setup proper monitoring and never get to the Real Issue to begin with. These sysadmin hacks are not helpful

Re: All my servers have an 8 GB empty file on disk

#154

Post by benhurmarcel »

SethTro wrote: post #31

Same idea as this game development legend https://www.dodgycoder.net/2012/02/coding-tricks-of-game-dev... > he had put aside those two megabytes of memory early in the development cycle. He knew from experience that it was always impossible to cut content down to memory budgets, and that many projects had come close to failing because of it. So now, as a regular practice, he always put aside a nice block of memory to…

https://thedailywtf.com/articles/The-Speedup-Loop

Re: All my servers have an 8 GB empty file on disk

#155

Post by ineedasername »

luckylion wrote: post #131
geocrasher wrote: post #124

For everyone saying "This isn't a real solution!" I'd like to explain why I think you're wrong. 1) It's not intended to be a Real Solution(tm). It's intended to buy the admin some time to solve the Real Issue. 2) Having a failsafe on standby such as this will save an admin's butt when it's 2am and PagerDuty won't shut up, and you're just awake enough to apply a temp fix and work on it in the morning. 3) Because "FIX…

> 1) It's not intended to be a Real Solution(tm). It's intended to buy the admin some time to solve the Real Issue. If you don't have monitoring, will you even be aware that your disk is filling up? If you do have monitoring, why are you artificially filling up your disk so that it will be at 100% more quickly instead of just setting your monitoring up to alert you when it's at $whateverItWasSetToMinusEightGB?

Because even if you have monitoring, some unforseen issue rapidly eating disk space at 3:00 am may not give you the time to solve it without downtime or degraded performance unless you can immediately remove the bottleneck while you troubleshoot.

Re: All my servers have an 8 GB empty file on disk

#157

Post by marricks »

luckylion wrote: post #131
geocrasher wrote: post #124

For everyone saying "This isn't a real solution!" I'd like to explain why I think you're wrong. 1) It's not intended to be a Real Solution(tm). It's intended to buy the admin some time to solve the Real Issue. 2) Having a failsafe on standby such as this will save an admin's butt when it's 2am and PagerDuty won't shut up, and you're just awake enough to apply a temp fix and work on it in the morning. 3) Because "FIX…

> 1) It's not intended to be a Real Solution(tm). It's intended to buy the admin some time to solve the Real Issue. If you don't have monitoring, will you even be aware that your disk is filling up? If you do have monitoring, why are you artificially filling up your disk so that it will be at 100% more quickly instead of just setting your monitoring up to alert you when it's at $whateverItWasSetToMinusEightGB?

Monitors can fail, you can miss an email, etc etc etc

There's always a big gap between what should never happen because you planned well and what does happen

Re: All my servers have an 8 GB empty file on disk

#158

Post by cmeacham98 »

londons_explore wrote: post #120

Linux has this built in... By default, only root can use the last 5% of disk space. That means you can fire up a root shell and know you have a buffer of free space to resolve the issue.

I believe this is an ext{2,3,4} feature. Unsure if it exists on btrfs, zfs, etc.

Re: All my servers have an 8 GB empty file on disk

#159

Post by lostcolony »

tshaddox wrote: post #99

How is this better than sounding alarms when free disk space drops below 8GB? If you’re going to ignore the alarms, then you’re going to have the same problem after you remove your spacer file and the disk fills up again!

It isn't either/or. It's very likely both.

Re: All my servers have an 8 GB empty file on disk

#160

Post by CrLf »

This is why the invention of LVM was such a good idea even for simpler systems (where some people claimed it was useless overhead). In my old sysadmin days I never allocated a full disk. The "menace" of an almost full filesystem was usually enough to incentivize cleanups but, when necessity came, the volume could be easily expanded.

I guess a big file is not a bad idea either.

Post reply on HN