Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

DwarFS: A fast high compression read-only file system

github.com

Post reply on HN
Print view View original

91–100 of 112 posts

Re: DwarFS: A fast high compression read-only file system

#91

Post by spaetzleesser »

Cojen wrote: post #30
pjc50 wrote: post #19

Earlier quoted context omitted.

I believe NT file compression works like this, and before that MSDOS "DriveSpace" ...

NTFS requires that files be manually converted to the compressed format. They're uncompressed in parts as requested, but this is only kept in RAM. I'm not aware of any built-in background task that converts files to/from the compressed format.

You can set the "Compressed" flag of a folder and from then on everything in that folder will be compressed/decompressed transparently. I have most of my disk compressed that way and never have seen problems.

Re: DwarFS: A fast high compression read-only file system

#92

Post by Arnavion »

hachari wrote: post #33
tutfbhuf wrote: post #23

Earlier quoted context omitted.

Is Btrfs stable yet?

Btrfs is default in Fedora 33.

It's also been the default for the root filesystem in openSUSE since 2012. It also integrates with the package manager - zypper creates pre- and post- snapshots when installing updates, and the snapper utility makes it easy to roll back if your system is busted after installing an update. It's saved my bacon on multiple occasions.

(btrfs also used to be the default for /home, though that changed at some point. When I made a new install last year the installer suggested xfs by default.)

Re: DwarFS: A fast high compression read-only file system

#93

Post by storedbox »

slagfart wrote: post #68

Perhaps not strictly on-topic, but is there any equivalent FS/program in Windows that will allow users to have read-only access to files that are deduplicated in some way? My use case is the MAME console archives, which are now full of copies of games from different localisations with 99% identical content. 7Z will compress them together and deduplicate, but breaks once the archive exceeds a few gigs. These archives…

You could use a [WIM image][1]. They can be mounted rw or ro and have file-level deduplication. Microsoft's official tooling is necessary to mount them on Windows as [the only open source implementation I am aware of][2] uses FUSE for mounting.

[1]: https://en.wikipedia.org/wiki/Windows_Imaging_Format [2]: https://wimlib.net/

Re: DwarFS: A fast high compression read-only file system

#94

Post by TimTheTinker »

mhx77 wrote: post #52
fefe23 wrote: post #17

It looks like the benefit is some kind of block or file deduplication. @OP: Can you please explain why you keep 50 gigs of perl around? :-) I use compressed read-only file systems all the time to save space on my travel laptop. I have one squashfs for firefox, one for the TeX base install, one for LLVM, one for qemu, one for my cross compiler collection. I suspect the gains over squashfs will be far less pronounced t…

> @OP: Can you please explain why you keep 50 gigs of perl around? :-) Sure. I've been the maintainer of a perl portability module (Devel::PPPort) for a long time and every release was tested against basically every possible version (and several build flag permutations) of perl that was potentially out in the wild.

Very impressive, to say the least.

(Not meant sarcastically :-)

Re: DwarFS: A fast high compression read-only file system

#95

Post by Reelin »

ggm wrote: post #82
ed25519FUUU wrote: post #63

I noticed that enabling compression on zfs made a huge difference with the source size of some of my largely text file petitions. I never turned on deduplication because I don’t want to bother with the memory overhead, but I bet that would help even further.

Most ZFS howto's now recommend against dedup on the prolongued memory cost consequences. Yes, you would get some block level compression outcome. But, you enter the cost/benefit hell of balancing CPU and memory at runtime.

Can't you periodically run the dedup out of band (for example whenever you scrub)? https://btrfs.wiki.kernel.org/index.php/Deduplication

Re: DwarFS: A fast high compression read-only file system

#96

Post by Izkata »

londons_explore wrote: post #54
mhx77 wrote: post #7

Earlier quoted context omitted.

Author here :) I'll add more benchmarks, this is still WIP and so far I've mainly tried to satisfy my own needs. My intention with DwarFS wasn't to write "a better SquashFS", but to make it better in certain scenarios (huge, highly redundant data) than SquashFS. SquashFS still has the big advantage of being part of the kernel, which makes it a lot more attractive for things like root file systems.

Are there git filesystems? If so, they could be a good comparison point too - gits PACK file format is pretty magic...

Apparently yes: https://github.com/presslabs/gitfs

Re: DwarFS: A fast high compression read-only file system

#97

Post by st_goliath »

Circa 2 years ago, I was working on a side project and got so annoyed with SquashFS tooling, that I decided to fix it instead. After getting stuck with the spaghetti code behind mksquashfs, I decided to start from scratch, having learnt enough about SquashFS to roughly understand the on-disk format.

Because squashfs-tools seemed pretty unmaintained in late 2018 (no activity on the official site & git tree for years and only one mailing list post "can you do a release?" which got a very annoyed response) I released my tooling as "squashfs-tools-ng" and it is currently packaged by a hand full of distros, including Debian & Ubuntu.[1]

I also thoroughly documented the on-disk format, after reverse engineering it[2] and made a few benchmarks[3].

For my benchmarks I used an image I extracted from the Debian XFCE LiveDVD (~6.5GiB as tar archive, ~2GiB as XZ compressed SquashFS image). By playing around a bit, I also realized that the compressed meta data is "amazingly small", compared to the actual image file data and the resulting images are very close to the tar ball compressed with the same compressor settings.

I can accept a claim of being a little smaller than SquashFS, but the claimed difference makes me very suspicious. From the README, I'm not quite sure: Does the Raspbian image comparison compare XZ compression against SquashFS with Zstd?

I have cloned the git tree and installed dozens of libraries that this folly thingy needs, but I'm currently swamped in CMake errors (haven't touched CMake in 8+ years, so I'm a bit rusty there) and the build fails with some still missing headers. I hope to have more luck later today and produce a comparison on my end using my trusty Debian reference image which I will definitely add to my existing benchmarks.

Also, is there any documentation on how the on-disk format for DwarFS and it's packing works which might explain the incredible size difference?

[1] https://github.com/AgentD/squashfs-tools-ng

[2] https://github.com/AgentD/squashfs-tools-ng/blob/master/doc/...

[3] https://github.com/AgentD/squashfs-tools-ng/tree/master/doc

Re: DwarFS: A fast high compression read-only file system

#98

Post by deepstack »

dj_mc_merlin wrote: post #9

> I started working on DwarFS in 2013 and my main use case and major motivation was that I had several hundred different versions of Perl that were taking up something around 30 gigabytes of disk space, and I was unwilling to spend more than 10% of my hard drive keeping them around for when I happened to need them. It fills me with joy that someone has been coding a fs for 7 years due to perl installs taking too much…

nice, wonder how this compare with MongoDB compression of files and objects. Seems like a great foundation for archiving data.

Re: DwarFS: A fast high compression read-only file system

#99

Post by mhx77 »

st_goliath wrote: post #97

Circa 2 years ago, I was working on a side project and got so annoyed with SquashFS tooling, that I decided to fix it instead. After getting stuck with the spaghetti code behind mksquashfs, I decided to start from scratch, having learnt enough about SquashFS to roughly understand the on-disk format. Because squashfs-tools seemed pretty unmaintained in late 2018 (no activity on the official site & git tree for years a…

This is really cool, I'll give squashfs-tools-ng a try!

> Does the Raspbian image comparison compare XZ compression against SquashFS with Zstd?

That's correct. It's not an exhaustive matrix of comparisons.

> Also, is there any documentation on how the on-disk format for DwarFS and it's packing works which might explain the incredible size difference?

The format as of 0.2.0 is actually quite simple. It's a list of compressed data blocks, followed by a metadata block (and a schema describing the metadata block). The metadata format is implemented by and documented in in [1].

There are probably 3 things that contribute to compression level:

1) Block size. DwarFS can use arbitrary block sizes (artificially limited to powers of two), and uses a much larger block size (16M) by default. SquasFS doesn't seem to be able to go higher than 1M.

2) Ordering files by similarity.

3) Segment deduplication. If segments of files overlap with previously seen data, these segments are referenced instead of written again. The minimum size of these segments can be configured and defaults to 2k. For my primary use case, of the 47.6 GB of input data, 28.2 GB are saved by file-level deduplication, and another 12.4 GB by this segment-level deduplication. So before the "real" compression algorithms actually kick in, there are only 7 GB of data left. As these are ordered by similarity, and stored in rather big blocks, some of the 16M blocks can actually be compressed down to less then 100k.

[1] https://github.com/mhx/dwarfs/blob/main/thrift/metadata.thri...

Re: DwarFS: A fast high compression read-only file system

#100

Post by rurban »

So I tried it out on my 17BG of perl builds. (just on my laptop, not on my big machine).

mkdwarfs crashed with recursive links (1-level, just pointing to itself) and when I removed dirs while running mkdwarfs, which were part of of the input path. Which is fair, I assume.

Post reply on HN