Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

All my servers have an 8 GB empty file on disk

brianschrader.com

Post reply on HN
Print view View original

471–480 of 715 posts

Re: All my servers have an 8 GB empty file on disk

#471

Post by efficax »

njovin wrote: post #335
victor9000 wrote: post #163

Earlier quoted context omitted.

Perfect example of the tragedy of the commons. If individuals don't create these balloon files then they won't be able to use the file server when they need it, yet by creating these balloon files the collective action depletes the shared resource of its main function.

This is similar to how some government agencies retain their budgets. At the end of the budget period they've only spent 80% of their allocated budget, so they throw out a bunch of perfectly good equipment/furniture/etc. and order new stuff so that their budget doesn't get cut the following year, rather than accepting that maybe they were over-budgeted to begin with. Rinse, repeat, thus continuing the cycle of wastin…

The departments of every large org do this, private sector and public

Re: All my servers have an 8 GB empty file on disk

#472

Post by _carbyau_ »

JustARandomGuy wrote: post #456
_carbyau_ wrote: post #272

Earlier quoted context omitted.

I noticed a weird thing here. All the regular pasta rapidly disappeared. Rice also was gone. But "Gluten Free"(I am a GFer) pasta - in this case, pasta made from rice - was fine! I amused myself with the idea that people:"would rather starve to death than eat that GF crap." :-)

Out of curiosity: did people really experience empty grocery shelves or are phrases such as "rapidly disappeared" meant more as hyperbole? I live in a Chicago suburb, and while variety did decrease (I still can't get Coke Zero Cherry) the basics were always in stock throughout the pandemic time.

I am in Melbourne Australia. Yeah, the normal pasta and rice disappeared for about a week, the shelve were literally empty.

But as mentioned, my GF pasta was fine.

And I don't think anyone really starved as most of the shelves were full of something. Just reduced options thanks to hoarding.

Re: All my servers have an 8 GB empty file on disk

#473

Post by herpderperator »

The author seems to forget that ext-based filesystems keep 5% of disk space available for root at all times by default, known as "reserved blocks".[0] That means if a non-root user uses all of the available space, it wasn't really all of the space -- root still has access to 5% free space within that partition. That's exactly the same as the useless 8GB file but in an officially-supported manner. If you run out of disk space, you actually have 5% left for root. So log in as root and fix the issue. Simple.

Also:

> On Linux servers it can be incredibly difficult for any process to succeed if the disk is full. Copy commands and even deletions can fail or take forever as memory tries to swap to a full disk and there's very little you can do to free up large chunks of space.

Why would memory swap to disk when the disk is full? I feel like the author is conflating potential memory pressure issues with disk issues.

How many serious production-grade servers even use swap, which usually just causes everything to grind to a halt if memory becomes full?

[0] https://ma.ttias.be/change-reserved-blocks-ext3-ext4-filesys...

Re: All my servers have an 8 GB empty file on disk

#474

Post by axiolite »

tytso wrote: post #175
birdyrooster wrote: post #47

Earlier quoted context omitted.

Funny because I have always tune2fs -m1 or tune2fs -m0 because the reserved space was never supposed to scale linearly with hard drive capacities and is not useful to userspace in anyway. Have never had any issues and been doing it for decades in commercial applications. In some cases, where you probably shouldn't be using ext3/4 anyways, we are talking about reclaiming TBs of reserved space. It's important to note t…

If you try to use a file system to 99% full --- and it doesn't matter whether it is a 10GB file system or a 10TB file system, you will see significant performance penalties as the file system gets badly fragmented. So that's why having a fixed percentage even for massively big disks still makes sense. Disk space is cheap enough that even 5% of a 14TB disk is really not that much money --- and if you see bad performan…

> If you try to use a file system to 99% full --- and it doesn't matter whether it is a 10GB file system or a 10TB file system, you will see significant performance penalties as the file system gets badly fragmented.

Not true, I've checked. I have plenty of Linux ext3 servers running for many years that routinely drop down to 1% free space for extended periods before being cleaned-up, which still have essentially zero fragmentation. You can create plenty of 10MB files on a multi-terabyte volume that has under 1% free space, as that's still tens of gigabytes to work with.

Obviously at some point you'll hit a severe problem and it's best to avoid taking a chance, but a fixed percentage really isn't the best measurement to tell you where that horizon will be.

Re: All my servers have an 8 GB empty file on disk

#475

Post by loonster »

JustARandomGuy wrote: post #456
_carbyau_ wrote: post #272

Earlier quoted context omitted.

I noticed a weird thing here. All the regular pasta rapidly disappeared. Rice also was gone. But "Gluten Free"(I am a GFer) pasta - in this case, pasta made from rice - was fine! I amused myself with the idea that people:"would rather starve to death than eat that GF crap." :-)

Out of curiosity: did people really experience empty grocery shelves or are phrases such as "rapidly disappeared" meant more as hyperbole? I live in a Chicago suburb, and while variety did decrease (I still can't get Coke Zero Cherry) the basics were always in stock throughout the pandemic time.

Yes, I have pictures of empty shelves on my phone.

Part of the problem was state government response. After each further restriction, people panic bought more.

Re: All my servers have an 8 GB empty file on disk

#478

Post by dilyevsky »

akdma wrote: post #465
dilyevsky wrote: post #430

Earlier quoted context omitted.

That’s not perfect world this is standard industry practice to have host level monitoring

THANK YOU. How are so many people in this thread content with saying “monitoring isn’t perfect, this solution is ingenious”. Ofc nothing is perfect and even when you do everything right things can still go wrong, but if you don’t have a ROBUST monitoring/alert system in place then you’re not even doing the bare minimum. They’re acting like it’s rocket science to set thresholds, and have meaningful alerts and checks i…

Yeah it’s crazy. If someone does this on their homelab server it’s probably fine but if they run it in production I really want to know because Im not buying jack from them.

Re: All my servers have an 8 GB empty file on disk

#479

Post by jabroni_salad »

dilyevsky wrote: post #153
geocrasher wrote: post #124

For everyone saying "This isn't a real solution!" I'd like to explain why I think you're wrong. 1) It's not intended to be a Real Solution(tm). It's intended to buy the admin some time to solve the Real Issue. 2) Having a failsafe on standby such as this will save an admin's butt when it's 2am and PagerDuty won't shut up, and you're just awake enough to apply a temp fix and work on it in the morning. 3) Because "FIX…

Setup proper monitoring and never get to the Real Issue to begin with. These sysadmin hacks are not helpful

Keep in mind that sysadmins are essentially babysitting software that they do not develop. The hacks that we come up with are to work around responsible party and help us get a good night's sleep instead of a 2am wakeup call. I try to cut you guys some slack, usually this proliferates when management decides they are willing to accept some inefficiency in favor of getting new features out the door. I get it, really.

My org is in the middle of a SRE introduction and for some reason I'm getting a lot of pushback on the topic of 'error budgets' and what to do with alerts when they are exceeded. Can't imagine why.

Re: All my servers have an 8 GB empty file on disk

#480

Post by [deleted] »

herpderperator wrote: post #473

The author seems to forget that ext-based filesystems keep 5% of disk space available for root at all times by default, known as "reserved blocks".[0] That means if a non-root user uses all of the available space, it wasn't really all of the space -- root still has access to 5% free space within that partition. That's exactly the same as the useless 8GB file but in an officially-supported manner. If you run out of di…

[deleted]
Post reply on HN