Live data from Hacker News

Harddrive Is Probably Full

marginalia.nu

31–40 of 49 posts

Re: Harddrive Is Probably Full

#31

... and It is ok. I bought 16TB disk because I needed that much storage. If I were using, say, only 10TB, I would be wasting 6TB. I can expand my LVM as much as I can. But I definitely want my disk to be full. Same goes for my RAM. And I want my CPU ide time to stay as close as possible to 0% and my load average as close as possible to my no. of CPUs. Anything else would mean oversizing and overspending.

[deleted]

Re: Harddrive Is Probably Full

#32

... and It is ok. I bought 16TB disk because I needed that much storage. If I were using, say, only 10TB, I would be wasting 6TB. I can expand my LVM as much as I can. But I definitely want my disk to be full. Same goes for my RAM. And I want my CPU ide time to stay as close as possible to 0% and my load average as close as possible to my no. of CPUs. Anything else would mean oversizing and overspending.

Slack has value, though. Your full disk is likely to be very fragmented, making file access slow. A CPU running a full throttle cannot deal with spikes in demand. full RAM means less cache space available. And these issues get asymptotically worse as you approach 100% utilization (which can have catastrophic effects in organizations which are too heavily optimized for high utilization).

Re: Harddrive Is Probably Full

#33

If you're on Windows, WizTree ( https://diskanalyzer.com/ ) is great for figuring out what's consuming your storage space. On Linux/macOS you can use `du` from the command line (or ask an AI to make a convenience wrapper for you).

Yes, I always use `du . -h 2>/dev/null | sort -k1hr | head -nN` where that last N is however many directories I want to show. I run this regularly and have for a very long time precisely because of the issue that the post talks about.

Re: Harddrive Is Probably Full

#34

The problem with filling up a ssd is that it writes and rewrites the same blocks over and over defeating the built in wear leveling. Filling an ssd creates the one situation where nand wear out is a problem.

Won't modern wear leveling migrate rarely written data to spread new writes out to its blocks?

Re: Harddrive Is Probably Full

#35
post #5

I recently ran `df` and was surprised to see that I'd used more than half of the two terabyte SSD in my laptop. So, being apparently atypically proactive, my first action was delete all of the target directories in my Rust projects and was equally surprised to see that it freed up nearly 10% of my disk usage (~200 GB)!

Are there many tiny files? They may take 4KiB of space each even though they're less than a KiB big (https://en.wikipedia.org/wiki/Disk_sector).

On a SanDisk portable SSD I own, the cluster size is even 4MiB. If you put a node modules directory in there with thousands of tiny files, it can "take up" several GiB of space.

Re: Harddrive Is Probably Full

#36
post #22

Earlier quoted context omitted.

> 46x faster than windirstat Thats a pretty bold claim, but is it really that much faster?

I just ran it for the first time... I've run windirstat many times including in the last two weeks and yes, this is very fast. According to its timing it took 3.63 seconds to scan my drive... I'm not sure if it really took almost 3 minutes for windirstat, but it did take a long time, so it's plausible. More importantly, ~ 4 seconds is much easier to wait for.

windirstat currently doesn't take 3min to index 500k+ files on an 8x8tb drive raidz2 that's 90% full. When I stopped using windoratat 3ish years ago it took a few minutes to index the same array.

I think you might he using an old version, or have an issue with your storage.

Re: Harddrive Is Probably Full

#37
post #5

I recently ran `df` and was surprised to see that I'd used more than half of the two terabyte SSD in my laptop. So, being apparently atypically proactive, my first action was delete all of the target directories in my Rust projects and was equally surprised to see that it freed up nearly 10% of my disk usage (~200 GB)!

Are there many tiny files? They may take 4KiB of space each even though they're less than a KiB big ( https://en.wikipedia.org/wiki/Disk_sector ). On a SanDisk portable SSD I own, the cluster size is even 4MiB. If you put a node modules directory in there with thousands of tiny files, it can "take up" several GiB of space.

This can be an important reason to enable compression (on some file systems it “solves” the problem) and why virtual machine file format can be important.

Re: Harddrive Is Probably Full

#38
post #29

Asked Claude to look at my full disk. Thought for 1 minute It found 114GB of Hugging Face models, 17GB worth of models in an obscure app I tested once. In places I would've never thought to look. I'd never have never found these myself. Problem solved. For now!

Just looking with ncdu/du or another tool which sorts folders by size would have been much faster and easier, was there any need to use Claude here?

It is so wild that someone would let Claude index all of the files on their computers for such a bane request. We are really turning into the people on WALL·E

Re: Harddrive Is Probably Full

#39
post #29

Asked Claude to look at my full disk. Thought for 1 minute It found 114GB of Hugging Face models, 17GB worth of models in an obscure app I tested once. In places I would've never thought to look. I'd never have never found these myself. Problem solved. For now!

Just looking with ncdu/du or another tool which sorts folders by size would have been much faster and easier, was there any need to use Claude here?

When your only tool is a hammer everything looks like a nail.

Re: Harddrive Is Probably Full

#40

The problem with filling up a ssd is that it writes and rewrites the same blocks over and over defeating the built in wear leveling. Filling an ssd creates the one situation where nand wear out is a problem.

The SSD firmware is smart enough to rotate the hot sectors over the entire space gradually.

It is a problem to have a full SSD from a performance perspective. Modern SSDs use pSLC (psuedo-single-level-cells) because they can write much faster. When you write to an SSD, you generally write in pSLC mode first, and once the data goes cold, it will get written back to MLC/TLC storage.

Once you are very full, pSLC gets disabled or becomes less effective, affecting write speeds.

There is also an effect known as write amplification that is exacerbated when a disk is nearly full. You want to write 4kb, but the available erasure zones are all full (either valid or invalidated sectors), so now the SSD has to erase a 4MB zone and compact the data there (along with your new 4kb) to write your 4kb.

This gets even worse when you have a filesystem that isn't SSD optimized, such as EXT4. It will cause further write amplication.

Keep your SSD's <80% full if you want good performance.

Post reply on HN