All my servers have an 8 GB empty file on disk • Hacker News Board

Re: All my servers have an 8 GB empty file on disk

#621

Post by kevincox » Fri, Mar 26, 2021, 1:21 PM UTC

tshaddox wrote: post #327

ineedasername wrote: post #155

Earlier quoted context omitted.

Because even if you have monitoring, some unforseen issue rapidly eating disk space at 3:00 am may not give you the time to solve it without downtime or degraded performance unless you can immediately remove the bottleneck while you troubleshoot.

Then why not automate the removal of the 8 GB spacer file when the disk gets full? Or in other words, just sound your alarms when there is 8 GB of free disk space.

Because if it is a broken process then it will fill up the disk again before you wake up and look at it.

I think the idea is that once you are at the system you can try to find out the cause without removing the file, or worse case remove the file and act fast (you may be on a short timer at this point). So for example if you find out that process X broke and is writing a ton of logs you can disable that process, remove the file, then most of your system is operational while you can properly fix the root cause or at the very least decide how to handle the data that filled up the disk in the first place. (You can't always just delete it without thought)

I think a more refined approach would be disk quotas that ensured that root (or a debugging user) always had a buffer to do the repairs. This file just serves as a system wide disk quota (but you need to remove it to take advantage of that reserved space).

Re: All my servers have an 8 GB empty file on disk

#622

Post by wp381640 » Fri, Mar 26, 2021, 1:29 PM UTC

anonu wrote: post #485

wp381640 wrote: post #442

Earlier quoted context omitted.

Lotus to Exchange migrations were all likely in the pre-Sarbanes Oxley and other retention regular era of email retention requirements iirc at the time the only industries that required retention were health, legal and government With SOX (PCI, FDIC, et al) retention laws we had another explosion of work rolling out all the compliance features of Exchange Those were crazy times getting everybody either migrated with…

I was using lotus at one of the largest banks in 2015... So no. Not pre sox.

JP Morgan? They’re infamous for bad internal IT

Re: All my servers have an 8 GB empty file on disk

#623

Post by dfxm12 » Fri, Mar 26, 2021, 1:29 PM UTC

Cullinet wrote: post #422

DevKoala wrote: post #266

Earlier quoted context omitted.

In California you pay ~1.5% in property taxes per year. Owning property is not cheap. In fact you never really own it.

I'm a brit who was only discouraged from the process of becoming a American (when much younger and on a terrific upwards trajectory which crazy events eventually skewered) and I would be hurt particularly by property tax now and actually the moment my income ceased growing. I can't remember if Florida doesn't have property tax and that then may explain why so many of my family retired there, but everywhere I wanted t…

If that's all that's keeping you back, you should check if the city/state you wish to live in has what's usually called a homestead exemption. Many places will reduce property tax if you meet a few criteria. The criteria varies, but it usually revolves around being on a fixed income and living on the property.

Florida has property tax, but they don't have an estate tax, which probably influences people to retire there...

Re: All my servers have an 8 GB empty file on disk

#624

Post by abduhl » Fri, Mar 26, 2021, 1:32 PM UTC

littlestymaar wrote: post #579

ike77 wrote: post #571

Earlier quoted context omitted.

For two reasons: 1) separation of duty: you might not be the best department to invest surplus 2) cost effectiveness: if you're operating with a deficit, as is generally the case with governments these days, this money is not free, so it could effectively be cheaper to give it back and re-borrow it when you actually need it

> 1) separation of duty: you might not be the best department to invest surplus But with this reasoning there is no surplus, because departments will spend their money at all cost. > 2) cost effectiveness: if you're operating with a deficit, as is generally the case with governments these days, this money is not free, so it could effectively be cheaper to give it back and re-borrow it when you actually need it That's…

Top management doesn’t “owe” them any money when they need it later. Say you budget $100 for dinner tonight and you go out and it costs $75. Do you owe the restaurant $25? While certainly some people might roll the $25 into the next day’s meals, some people might allocate that $25 to another cost center like buying a new car.

Budgets are meant to estimate costs and manage cash flow. From a greedy team perspective it’s best (and self interested) to try to game the system as much as possible so you get the largest share of the pie. From the organizational perspective it’s best to reallocate capital efficiently, especially if a team consistently over budgets.

Re: All my servers have an 8 GB empty file on disk

#625

Post by herpderperator » Fri, Mar 26, 2021, 1:34 PM UTC

Stevvo wrote: post #606

herpderperator wrote: post #473

The author seems to forget that ext-based filesystems keep 5% of disk space available for root at all times by default, known as "reserved blocks".[0] That means if a non-root user uses all of the available space, it wasn't really all of the space -- root still has access to 5% free space within that partition. That's exactly the same as the useless 8GB file but in an officially-supported manner. If you run out of di…

Sure, but one of the most common causes of disks being filled on Linux is either the kernel or some process running as root filling the disk with endless repeating log entries.

It's a good point that the process that is flooding the disk could be running as root and therefore bypasses the reserved blocks restriction.

It's also worth noting that the kernel is not a process and it does not write directly to files in the conventional sense. syslog-ng or its equivalents (which do run as root) will pick up messages from the kernel's circular ring buffer at /dev/kmsg and write them to a text file such as /var/log/kern.log, so it's possible that the kernel or one of its modules are verbose enough that syslog-ng causes that text file to get big. However, these log files can be limited to a certain size and/or logrotated[0] daily which will also remove log files older than a configured amount of time. In other words, there are better ways to manage servers than creating 8GB files as a bandaid.

[0] https://linux.die.net/man/8/logrotate

Re: All my servers have an 8 GB empty file on disk

#626

Post by berkes » Fri, Mar 26, 2021, 1:39 PM UTC

dilyevsky wrote: post #275

berkes wrote: post #260

Earlier quoted context omitted.

"proper monitoring" is extremely broad. And, I would say, almost unreachable goal. You have it mail you when it goes over 80% disk usage (and what if you are on holiday)? Does it mail all colleagues? Who picks it up (I thought Bob picked it up, but Bob thought Anne picked it up. So no one did)? Does it come and wake you in person when it reaches 92%? Will this catch this async job that fails (but should never) in an…

Slack warning/ticket at 75%, page at 85% (to oncall obviously). Don’t let user workload crap into your root partition. I’ve been doing this for over 10 years and managed many thousands of nodes and literally don’t recall full disk problem unless it was in staging somewhere where monitoring was deliberately disabled.

Your requirements for "proper monitoring" are not everyones requirements.

On a current gig, we host at heroku. Our monitoring is all about 95th percentile response-times, secondary services, backlogs, slow-queries and whatnots. For another job, "disk space filling up" is important. Again another job will need to monitor email-delivery-rates and so on and so forth.

Re: All my servers have an 8 GB empty file on disk

#627

Post by throwaway0a5e » Fri, Mar 26, 2021, 1:41 PM UTC

bottled_poe wrote: post #534

Aeolun wrote: post #383

Earlier quoted context omitted.

I think the problem is that you do not need 100% of your budget every year, but getting it back when you do need it is much harder than keeping it in the first place. Definite case of misaligned incentives.

Why aren’t there rolling budgets over multiple years with annual reviews on spend?

Because nobody in government has any incentive to do something that risks shrinking the size of the pie.

Re: All my servers have an 8 GB empty file on disk

#628

Post by throwaway0a5e » Fri, Mar 26, 2021, 1:44 PM UTC

duxup wrote: post #363

njovin wrote: post #335

Earlier quoted context omitted.

This is similar to how some government agencies retain their budgets. At the end of the budget period they've only spent 80% of their allocated budget, so they throw out a bunch of perfectly good equipment/furniture/etc. and order new stuff so that their budget doesn't get cut the following year, rather than accepting that maybe they were over-budgeted to begin with. Rinse, repeat, thus continuing the cycle of wastin…

Government? I see that as the SOP for large companies too...

True,

But large companies tend to have MBA types scurrying around rooting this stuff out as it pops up or shortly thereafter. Government has no such sort of immune system to fix these problems on the go. It just gets sicker and sicker until the tax payers vote for something drastic or revolt.

You see this in nonprofit entities too. They get big, abstract away from their mission and waste a lot of money until someone gets tasked with cleaning house or a more mission-driven comes along and replaces them.

Re: All my servers have an 8 GB empty file on disk

#629

Post by cmsj » Fri, Mar 26, 2021, 1:47 PM UTC

geocrasher wrote: post #124

For everyone saying "This isn't a real solution!" I'd like to explain why I think you're wrong. 1) It's not intended to be a Real Solution(tm). It's intended to buy the admin some time to solve the Real Issue. 2) Having a failsafe on standby such as this will save an admin's butt when it's 2am and PagerDuty won't shut up, and you're just awake enough to apply a temp fix and work on it in the morning. 3) Because "FIX…

> 1) It's not intended to be a Real Solution(tm). It's intended to buy the admin some time to solve the Real Issue.

It doesn't do that though. If you don't have monitoring/alerting that can either a) give you sufficient notice that you're trending out of disk space, b) take action on its own (e.g. defensively shutting down the machine), or c) both of the above, then having your server disks fill up is bad whether you have a ballast file or not.

If your database server goes to 100%, you can't trust your database anymore whether you could ssh in and delete an 8GB file or not.

That's it.

Re: All my servers have an 8 GB empty file on disk

#630

Post by thesmok » Fri, Mar 26, 2021, 2:03 PM UTC

names_are_hard wrote: post #273

quesera wrote: post #217

Earlier quoted context omitted.

Most motorcycles are surprisingly manual. This was originally a necessity (like in cars), but remains aesthetically preferable for many riders. OTOH, Honda Goldwings have stereo systems. They might grow an automatic fuel reserve switcher-backer someday too. :)

Many new bikes come with a lot of rider aids for safety (ABS, TCS) as well as all kinds of electronics (fuel maps), so this is changing. But of course manual transmission won't go away until bike are electric. I am one of those who likes things old school. My bike still has a carburetor, has no fuel light or tachometer, and I have certainly had some practice reaching down to turn the fuel petcock to reserve while spu…

> But of course manual transmission won't go away until bike are electric.

See multiple Honda bikes with DCT (dual clutch transmission). This is what I'm planning to get as my first bike.