Viewing profile — mhx77

FWIW, I've updated the comparison using the latest versions of EROFS and DwarFS. There definitely have been improvements to EROFS in the meantime. DwarFS is still orders of magnitu…

Thanks for the feedback! I was probably a little too excited when writing the documentation. Reading it again with some distance, I can certainly see why this might be a bit off-pu…

https://en.wikipedia.org/wiki/Locality-sensitive_hashing https://en.wikipedia.org/wiki/Nilsimsa_Hash https://github.com/mhx/dwarfs/blob/main/doc/mkdwarfs.md#nils...

Correct, there are no images in the data except for 68 PNGs. It's just HTML files.

It's ever so slightly smaller than a .tar.xz of the same data. The main difference being that you don't have to fully extract it in order to access the data.

That's definitely a fair point! DwarFS actually works perfectly fine on 64-bit ARM and would likely work fine on a smartphone as well. Still (and I might be completely wrong about …

I've got a (relatively old) snapshot of the English Wikipedia that I'm using for testing. The snapshot is around 200 GiB in 14,000,000 files and compresses down to an 11 GiB DwarFS…

https://github.com/mhx/dwarfs?tab=readme-ov-file#with-erofs It's been a while since I did that comparison, so the results could be significantly different now.

Somewhat related (and definitely born out of a very similar use case): https://github.com/mhx/dwarfs I initially built this for having access to 1000+ Perl installations (spanning …

> mkdwarfs crashed with recursive links (1-level, just pointing to itself) That's odd, it shouldn't crash with links at all, as it doesn't actively follow links. Can you please fil…

Yes, and I've documented this now: https://github.com/mhx/dwarfs/blob/main/doc/dwarfs.md#settin...

> have you investigated why this might be the case? Very briefly. It looks like clang has a different strategy breaking up the code (which is mostly C++ templates) into actual func…

> If you upload a module to CPAN, you automatically get it tested against a huge matrix of configurations. Very true, and it's definitely a great service! However, the set of versi…

This is really cool, I'll give squashfs-tools-ng a try! > Does the Raspbian image comparison compare XZ compression against SquashFS with Zstd? That's correct. It's not an exhausti…

Precisely.

Files that you've accessed will be kept in the kernel's cache. The cache I was talking about is a cache for decompressed blocks. Single files can stretch across multiple blocks, so…

> @OP: Can you please explain why you keep 50 gigs of perl around? :-) Sure. I've been the maintainer of a perl portability module (Devel::PPPort) for a long time and every release…

> How opposed would you be to this being reworked to being able to be mainline kernel support too? I don't see any way of getting this anywhere near the kernel without a full rewri…

Author here :) I'm not sure low-spec hardware is necessarily the best use case for DwarFS. It doesn't necessarily use less resources than SquashFS, although it can create file syst…

Hahaha, I haven't actually been coding on this for that long, it's more that I coded for a few weeks back in 2013 and only found the motivation to resurrect the whole thing a few w…

Read-only media, for example. Or in general, stuff that doesn't really change. In my case: https://github.com/mhx/dwarfs#history

Author here :) I'll add more benchmarks, this is still WIP and so far I've mainly tried to satisfy my own needs. My intention with DwarFS wasn't to write "a better SquashFS", but t…