DwarFS: A fast high compression read-only file system

Post by Cojen » Tue, Dec 01, 2020, 3:43 PM UTC

spaetzleesser wrote: post #91

Cojen wrote: post #30

Earlier quoted context omitted.

NTFS requires that files be manually converted to the compressed format. They're uncompressed in parts as requested, but this is only kept in RAM. I'm not aware of any built-in background task that converts files to/from the compressed format.

You can set the "Compressed" flag of a folder and from then on everything in that folder will be compressed/decompressed transparently. I have most of my disk compressed that way and never have seen problems.

That's cool, I hadn't thought of that. But does it identify "hot"/"cold" files that might benefit from being converted automatically to/from the compressed format? That would be a very nice feature to have.

Post by rurban » Tue, Dec 01, 2020, 5:21 PM UTC

mhx77 wrote: post #110

rurban wrote: post #100

So I tried it out on my 17BG of perl builds. (just on my laptop, not on my big machine). mkdwarfs crashed with recursive links (1-level, just pointing to itself) and when I removed dirs while running mkdwarfs, which were part of of the input path. Which is fair, I assume.

> mkdwarfs crashed with recursive links (1-level, just pointing to itself) That's odd, it shouldn't crash with links at all, as it doesn't actively follow links. Can you please file a bug if you can reproduce this? > and when I removed dirs while running mkdwarfs, which were part of of the input path I guess this is fair, but I'll try to take a look anyway. :-) > On success, mkdwarfs needed 1 hr, and reduced 219 dirs…

No, I'm fine with the high default compression rate. I only do this once a decade, and one hour is fair for this. A Fedora upgrade needs 5-8 hrs.

I have lot of -g info, because I use it mainly for debugging XS problems with old versions. The hashes change for each object, so reduplication is mostly only useful for source files. I really need high compression, which is the default.

Re: DwarFS: A fast high compression read-only file system

Re: DwarFS: A fast high compression read-only file system