Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

All my servers have an 8 GB empty file on disk

brianschrader.com

Post reply on HN
Print view View original

511–520 of 715 posts

Re: All my servers have an 8 GB empty file on disk

#511

Post by hunter2_ »

herbcso wrote: post #507
Aeolun wrote: post #383

Earlier quoted context omitted.

I think the problem is that you do not need 100% of your budget every year, but getting it back when you do need it is much harder than keeping it in the first place. Definite case of misaligned incentives.

I've never understood this about budgeting. So you allocate. A budget. These are fund YOU ARE PLANNING TO SPEND! So, OK, you DON'T spend them this year. Why the fuck don't you get to SAVE THAT MONEY!? No, instead you are punished for now spending it all and you cannot create a realistic budget for next year - why the hell not!? Sorry, but this frustrates the hell out of me! What am I missing here? What arcane bit of…

Just brainstorming, but maybe the irony is that your scenario somehow has even worse incentives? For example, building up that rollover number would gamify thriftiness greatly exceeding the typical "oops didn't spend it all," and think about the consequences of when you DIY something you're not the best at instead of hiring the pros.

Re: All my servers have an 8 GB empty file on disk

#512

Post by das4119 »

geocrasher wrote: post #124

For everyone saying "This isn't a real solution!" I'd like to explain why I think you're wrong. 1) It's not intended to be a Real Solution(tm). It's intended to buy the admin some time to solve the Real Issue. 2) Having a failsafe on standby such as this will save an admin's butt when it's 2am and PagerDuty won't shut up, and you're just awake enough to apply a temp fix and work on it in the morning. 3) Because "FIX…

I agree with this assessment. Of course its not a solution. Its delaying the inevitable. But depending on the rate of "filling up the disk for unknown reason" it will buy you time.

So when you're running out of space, you immediately delete the junk file. Suddenly there's "No Problem" and you've reset the symptom back to hopefully well before it was an issue. Now you can run whatever you need to, do reports, do traces etc. Even add more storage if necessary.

More importantly, as soon as you delete that junk file now you have space for logs. You have space and time for investigation.

Re: All my servers have an 8 GB empty file on disk

#513

Post by [deleted] »

herpderperator wrote: post #473

The author seems to forget that ext-based filesystems keep 5% of disk space available for root at all times by default, known as "reserved blocks".[0] That means if a non-root user uses all of the available space, it wasn't really all of the space -- root still has access to 5% free space within that partition. That's exactly the same as the useless 8GB file but in an officially-supported manner. If you run out of di…

[deleted]

Re: All my servers have an 8 GB empty file on disk

#514

Post by cellularmitosis »

tehjoker wrote: post #141
geocrasher wrote: post #124

For everyone saying "This isn't a real solution!" I'd like to explain why I think you're wrong. 1) It's not intended to be a Real Solution(tm). It's intended to buy the admin some time to solve the Real Issue. 2) Having a failsafe on standby such as this will save an admin's butt when it's 2am and PagerDuty won't shut up, and you're just awake enough to apply a temp fix and work on it in the morning. 3) Because "FIX…

This reminds me of the reserve tank toggle on some motorcycles. When you run out of gas, you switch the toggle and drive directly to a gas station.

buying a translucent gas tank has been one of the best investments I've made into my bike.

Re: All my servers have an 8 GB empty file on disk

#515

Post by hadsed »

Aeolun wrote: post #383
njovin wrote: post #335

Earlier quoted context omitted.

This is similar to how some government agencies retain their budgets. At the end of the budget period they've only spent 80% of their allocated budget, so they throw out a bunch of perfectly good equipment/furniture/etc. and order new stuff so that their budget doesn't get cut the following year, rather than accepting that maybe they were over-budgeted to begin with. Rinse, repeat, thus continuing the cycle of wastin…

I think the problem is that you do not need 100% of your budget every year, but getting it back when you do need it is much harder than keeping it in the first place. Definite case of misaligned incentives.

Seems it's the yearly cycle not matching up to the longer cycle of certain needs that's a problem.

Re: All my servers have an 8 GB empty file on disk

#516

Post by chousuke »

GekkePrutser wrote: post #60
chousuke wrote: post #45

Earlier quoted context omitted.

What servers usually run as root? Some may start as root, but usually drop privileges for the actual server processes quickly, eg. apache, nginx, sshd. Nothing that actually does the "serving" or accesses data should be running as root.

No but the logfile writers are usually running as root AFAIK. And this is what tends to fill up the disk.

My rule of thumb to avoid these issues is that application/server data gets its own dedicated volume that contains nothing else: logs get their own volume, and root its own. It's an especially bad idea for an application to put its data and logs in the same directory where its binaries reside.

That way, even if your log volume or root somehow fills up before monitoring had a chance to react, your service is unaffected. You can even catch issues pre-emptively by keeping log volumes small so that weird behaviour is likely to trigger an alert before anything goes truly wrong.

On cloud instances, it's silly to put anything on the instance root volume (on AWS, I keep them at the default 8 GB; it's never been a problem) when you can just attach an arbitrary number of additional disks. Container systems would use persistent volumes, and with physical servers, you use LVM or equivalent. This solves most disk allocation issues and makes operations easy when you need more space.

Re: All my servers have an 8 GB empty file on disk

#517

Post by alliao »

rigrassm wrote: post #380
jmkb wrote: post #351

Earlier quoted context omitted.

Had this in a 70s F150. A "Main - Aux" switch on the dash, right above the 8-track player. I used to let the main tank sputter out on fumes and then triumphantly shout "Rerouting auxiliary power to engine!" while sliding the switch. Letting them empty out alternately would have been a lot smarter.

My father drove a '95 F-150 for years that had the dual tanks. Shortly after highschool I got in accident that ended up totaling my vehicle and got a couple months I was using his truck (he runs an Auto repair shop from a garage behind the house so he almost always had something available to drive) and I ended up using it to go out on a date with someone I had met at work. I noticed on the way to pick them up that th…

thank you for the journey! (I must be getting old, loving these)

Re: All my servers have an 8 GB empty file on disk

#518

Post by ycombinete »

eevilspock wrote: post #447
victor9000 wrote: post #163

Earlier quoted context omitted.

Perfect example of the tragedy of the commons. If individuals don't create these balloon files then they won't be able to use the file server when they need it, yet by creating these balloon files the collective action depletes the shared resource of its main function.

Tragedy of the Commons is the libertarian "private property is essential" interpretation. It's a cynical take, assuming that human selfishness is the deepest of truths and that there is no use fighting it, that the best solution is to organize society around it. The conventional Game Theory take is that this is a prisoners dilemma, and everyone creating balloon.txt files are defecting. They are making the most ration…

It’s not just libertarians. I’m sure that even communists accept the premise of human self interest, but instead of private property their solution is for one all powerful government to own everything

Re: All my servers have an 8 GB empty file on disk

#519

Post by cellularmitosis »

herpderperator wrote: post #473

The author seems to forget that ext-based filesystems keep 5% of disk space available for root at all times by default, known as "reserved blocks".[0] That means if a non-root user uses all of the available space, it wasn't really all of the space -- root still has access to 5% free space within that partition. That's exactly the same as the useless 8GB file but in an officially-supported manner. If you run out of di…

> So log in as root and fix the issue. Simple.

"just fix the issue" often takes longer than 'rm spacer.img', which allows non-root processes to begin working immediately.

In practice, the 5% root reserve is not really an effective substitute for OP's technique.

Re: All my servers have an 8 GB empty file on disk

#520

Post by rabuse »

herpderperator wrote: post #473

The author seems to forget that ext-based filesystems keep 5% of disk space available for root at all times by default, known as "reserved blocks".[0] That means if a non-root user uses all of the available space, it wasn't really all of the space -- root still has access to 5% free space within that partition. That's exactly the same as the useless 8GB file but in an officially-supported manner. If you run out of di…

I didn't know this. Thanks!
Post reply on HN