Live data from Hacker News

All my servers have an 8 GB empty file on disk

brianschrader.com

491–500 of 715 posts

Re: All my servers have an 8 GB empty file on disk

#491
post #363

Earlier quoted context omitted.

Government? I see that as the SOP for large companies too...

It's possible that once a company reaches a certain size, it's inevitable. Corporations internally have the same top-down centralized organizational structure as a typical government. Market forces can't eliminate that kind of inefficiency if it invariably affects all large enterprises, and the economies of scale enjoyed by such companies outweigh the perverse incentives of sub-organizations. What strikes me as uniqu…

Also don’t forget how unevenly applied market forces are: if McDonald’s started charging 10% more for a hamburger they’d lose sales to Burger King a LOT faster than, say, Comcast or Oracle because the products are basically the same and most customers can switch almost effortlessly whereas you have to be especially mad to trench fiber out to your house or migrate every database in a large enterprise.

Any business with a natural monopoly, high migration costs, etc. can support a surprising amount of inefficiency even if most of their customers find the experience unsatisfying.

Re: All my servers have an 8 GB empty file on disk

#493
post #175

Earlier quoted context omitted.

If you try to use a file system to 99% full --- and it doesn't matter whether it is a 10GB file system or a 10TB file system, you will see significant performance penalties as the file system gets badly fragmented. So that's why having a fixed percentage even for massively big disks still makes sense. Disk space is cheap enough that even 5% of a 14TB disk is really not that much money --- and if you see bad performan…

> If you try to use a file system to 99% full --- and it doesn't matter whether it is a 10GB file system or a 10TB file system, you will see significant performance penalties as the file system gets badly fragmented. Not true, I've checked. I have plenty of Linux ext3 servers running for many years that routinely drop down to 1% free space for extended periods before being cleaned-up, which still have essentially zer…

It's true, but it's more true for some file systems than others. When you write a file larger than the contiguous available space after its starting point, a file system must break the file into "extents" (chunks). The less space available, the smaller the extents tend to be, and the more fragmentation you will impose for continued writes. It's just math.

Different file systems have wildly different strategies and data structures behind this process, however. Some drop to their knees over 92-93%. Some can write to the last byte with reasonable efficiency—but it'll never be as fast as when it was empty. Copy-on-write systems like ZFS tend to do poorly under near-full conditions.

Re: All my servers have an 8 GB empty file on disk

#495
Another trick you can use is to adjust the size of the FS tables, NFS4 can do this very quickly and free up space.

However on a sketchy drive this is obviously not a wise move.

Actually wait, NFS2/3/4 has reserved block counts you can free.

      # tune2fs -m 3 /dev/md2
      # Setting reserved blocks percentage to 2%

Re: All my servers have an 8 GB empty file on disk

#497

This reminds me of an old gamedev story that I have no idea how to find. The project was getting near to shipping, they had cut all the space they could cut, but they still needed another megabyte of space. After a week of this, the senior dev told the narrator to meet him in his office, and he closed the door. He opened one of the project files and deleted a 1 MB static array. "At the beginning of development I alwa…

https://www.gamasutra.com/view/feature/132500/dirty_coding_t...

Re: All my servers have an 8 GB empty file on disk

#498

Earlier quoted context omitted.

I suspect the blog author did not understand this (based on the content) - as a Linode user myself, I just had a look at one of my VMs and they install with the regular 5% reserved space (ext4/Debian).

Funny because I have always tune2fs -m1 or tune2fs -m0 because the reserved space was never supposed to scale linearly with hard drive capacities and is not useful to userspace in anyway. Have never had any issues and been doing it for decades in commercial applications. In some cases, where you probably shouldn't be using ext3/4 anyways, we are talking about reclaiming TBs of reserved space. It's important to note t…

It's reserved for the superuser, not the kernel

Re: All my servers have an 8 GB empty file on disk

#499

Earlier quoted context omitted.

> This is similar to how some government agencies retain their budgets. The non-government sector isn’t immune to this.

If whatever actions turn a non government entity into something inefficient, then the entity wont survive for long and will go out of business (or at least that's the hope of a competitive free market economy)

Hope is a really interesting way to frame something that has consistently failed to prove true after centuries of theory and decades of targeted policy changes.

Re: All my servers have an 8 GB empty file on disk

#500
post #383
post #335

Earlier quoted context omitted.

This is similar to how some government agencies retain their budgets. At the end of the budget period they've only spent 80% of their allocated budget, so they throw out a bunch of perfectly good equipment/furniture/etc. and order new stuff so that their budget doesn't get cut the following year, rather than accepting that maybe they were over-budgeted to begin with. Rinse, repeat, thus continuing the cycle of wastin…

I think the problem is that you do not need 100% of your budget every year, but getting it back when you do need it is much harder than keeping it in the first place. Definite case of misaligned incentives.

Yep! The problem happens when you divide the safety buffer up in the first place. Safety buffers demand to be shared, when one part does not use all of its safety margin you want to transfer that to another system.

Another surprising place where this happens is project scheduling. We budget time for each individual step of a project based on our guess of a 90% or 95% success rate, then our "old-timers' experience" kicks in and we double or triple our time for all the steps together, then our boss adds 50% before giving the estimate to their boss, which sounds gratuitous but it is to protect you because their boss looks at how grotesquely long the estimate is and barks out a cut of 20%, so the overall effect of those two is (3/2) × (4/5), so your boss still netted you a 20% buffer while making the skip-level feel very productive and important.

Say the 50%-confidence-to-95%-confidence gives you 30% more time as safety buffer, and you only double the estimate, and the work that you missed in your initial assessment, while it's not gonna be say half the project, maybe generously it's a third of the project or so. So the project actually takes time 1.5 measured properly, you have together budgeted 1.3 × 2 × 1.2 = 3.12 time. The total project deadline is more than half composed of safety buffer. And we still consistently overrun~!

But if Alice needs to work on some step after Bob, and Bob finishes early, when does Alice start on it? Usually not when Bob finishes. Alice has been told that Bob has until X deadline to complete, and has scheduled herself with other tasks until X. Bob says "I got done early!" and Alice says "that's great, I'm still working on other things but I will pick my tasks up right on time." Bob's safety buffer gets wasted. This does not always cause any impact to the deadline, but it does for the important steps.

Of course, if you are a web developer you already know this intuitively because you work on servers, and you don't run your servers (Alice, for example) at 100% load, because if you do then you can't respond to new requests (Bob's completion event) with low latency. It's worth thinking about, in an efficient workplace, how much are you not working so that you have excess capacity to operate efficiently?

Post reply on HN