Earlier quoted context omitted.
Is Btrfs stable yet?
I'm only ever switching to a new filesystem if its correctness has been formally verified.
For which filesystems is that case?
DwarFS: A fast high compression read-only file system
71–80 of 112 posts
Re: DwarFS: A fast high compression read-only file system
#72Perhaps not strictly on-topic, but is there any equivalent FS/program in Windows that will allow users to have read-only access to files that are deduplicated in some way? My use case is the MAME console archives, which are now full of copies of games from different localisations with 99% identical content. 7Z will compress them together and deduplicate, but breaks once the archive exceeds a few gigs. These archives…
s/ask/request/g
Re: DwarFS: A fast high compression read-only file system
#73Earlier quoted context omitted.
Files that you've accessed will be kept in the kernel's cache. The cache I was talking about is a cache for decompressed blocks. Single files can stretch across multiple blocks, so you need to be able to keep more than one in memory anyway. However, decompressed files are kept in the cache in the hope that further (or even concurrent) reads will access the same blocks. Taking the example from the README where over a…
Ah, I see. So this specifically saves the decompression time for data you've already decompressed, if another file references the same data?
Precisely.
Re: DwarFS: A fast high compression read-only file system
#74I wish there was a semi-compressed transparent filesystem layer which slowly compresses the least recently used files in the background, and un-compresses files upon use. That way you could store much more mostly unused content than space on the disk, without sacrificing accessibility.
I believe the term of art that applies here is "Hierarchical Storage Management". Along with automatically moving data between high-cost and low-cost storage media, the low-cost storage media for your filesystem of choice for the kind of compressing you described can simply be fast disk on a compressing filesystem.
Re: DwarFS: A fast high compression read-only file system
#75I'm curious, why do you have so many perl installations around. I thought I'd got a fair number of python venvs kicking around for each of the repos I'm dealing with, but nowhere near that many.
My Python shits have pip requirements that easily dump 3-4 gigs in a venv folder. Do that once or twice a month when starting a new project for a couple of years and it gets messy...
I'd like to see a pip freeze of whatever you're doing to consistently need venvs of that size.
Re: DwarFS: A fast high compression read-only file system
#76It looks like the benefit is some kind of block or file deduplication. @OP: Can you please explain why you keep 50 gigs of perl around? :-) I use compressed read-only file systems all the time to save space on my travel laptop. I have one squashfs for firefox, one for the TeX base install, one for LLVM, one for qemu, one for my cross compiler collection. I suspect the gains over squashfs will be far less pronounced t…
> @OP: Can you please explain why you keep 50 gigs of perl around? :-) Sure. I've been the maintainer of a perl portability module (Devel::PPPort) for a long time and every release was tested against basically every possible version (and several build flag permutations) of perl that was potentially out in the wild.
The single case in the known universe.
Re: DwarFS: A fast high compression read-only file system
#77I wish there was a semi-compressed transparent filesystem layer which slowly compresses the least recently used files in the background, and un-compresses files upon use. That way you could store much more mostly unused content than space on the disk, without sacrificing accessibility.
I don't know about you guys, but most of the stuff that takes up space on my drives are:
1) Videos from my DSLR
2) RAW images from my DSLR
3) Various movies / TV series I downloaded
4) Game files (most of which are textures and 3D models)
None of that stuff is really compressible.
Re: DwarFS: A fast high compression read-only file system
#78Perhaps not strictly on-topic, but is there any equivalent FS/program in Windows that will allow users to have read-only access to files that are deduplicated in some way? My use case is the MAME console archives, which are now full of copies of games from different localisations with 99% identical content. 7Z will compress them together and deduplicate, but breaks once the archive exceeds a few gigs. These archives…
You probably need to de-duplicate before compression, at least for many compression schemes.
Re: DwarFS: A fast high compression read-only file system
#79Perhaps not strictly on-topic, but is there any equivalent FS/program in Windows that will allow users to have read-only access to files that are deduplicated in some way? My use case is the MAME console archives, which are now full of copies of games from different localisations with 99% identical content. 7Z will compress them together and deduplicate, but breaks once the archive exceeds a few gigs. These archives…
If you are using windows server, data deduplication[1] is available on non-system volumes to do exactly this.
If you're using a Windows client, there is a way of enabling this, but it's not exactly supported, for a variety of reasons.
[1]https://docs.microsoft.com/en-us/windows-server/storage/data...
Re: DwarFS: A fast high compression read-only file system
#80Perhaps not strictly on-topic, but is there any equivalent FS/program in Windows that will allow users to have read-only access to files that are deduplicated in some way? My use case is the MAME console archives, which are now full of copies of games from different localisations with 99% identical content. 7Z will compress them together and deduplicate, but breaks once the archive exceeds a few gigs. These archives…
It's probably a hack, but you can try "backing up" your files with bup, restic or borg, and mount the resulting snapshot with FUSE