Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Mounting tar archives as a filesystem in WebAssembly

jeroen.github.io

Post reply on HN
Print view View original

21–30 of 42 posts

Re: Mounting tar archives as a filesystem in WebAssembly

#21

Post by mxmlnkn »

phiresky wrote: post #5

I'm a bit disappointed that this only solves the "find index of file in tar" problem, but not at all the "partially read a tar.gz" file problem. So really you're still reading the whole file into memory, so why not just extract the files properly while you are doing that? Takes the same amount of time (O(n)) and less memory. The gzip-random-access problem one is a lot more difficult because the gzip has internal stat…

> Apparently the internal state is only 32kB

Exactly. And often this state is either highly compressible or non-compressible but only sparsely used. The latter can then be made compressible by replacing the unused bytes with zeros.

Ratarmount uses indexed_gzip, and when parallelization makes sense, it also uses rapidgzip. Rapidgzip implements the sparsity analysis to increase compressibility and then simply uses the gztool index format, i.e., compresses each 32 KiB using gzip itself, with unused bytes replaced with zeros where possible.

indexed_gzip, gztool, and rapidgzip all support seeking in gzip streams, but all have some trade-offs, e.g., rapidgzip is parallelized but will have much higher memory usage because of that than indexed_gzip or gztool. It might be possible to compile either of these to WebAssembly if there is demand.

Re: Mounting tar archives as a filesystem in WebAssembly

#22

Post by crabique »

Very cool, I wish there were something similar to this for filesystem images though.

Just recently I needed to somehow generate a .tar.gz from a .raw ext4 image and, surprisingly, there's still no better option than actually mounting it and then creating an archive.

I managed to "isolate" it a bit with guestfish's tar-out, but still it's pretty slow as it needs to seek around the image (in my case over NBD) to get the actual files.

Re: Mounting tar archives as a filesystem in WebAssembly

#23

Post by QuantumNomad_ »

chungy wrote: post #20
johannes1234321 wrote: post #17

Earlier quoted context omitted.

> Uncompressed .tar is a reasonable choice for this application Yes, uncompressed tar (with transfer compression, which is offered in HTTP) is an option for some amount of data. Till the point where it isn't. zip has similar benefits as tar(+transfer compression) but a later point where it fails for such a scenario.

Zip allows you to set compression algorithm on a per-file basis, including no compression.

You can achieve the same with tar if you individually compress the files before adding them to the tar ball instead of compressing the tar ball itself.

I don’t see how that plus a small index of offsets would be notably more or less work to do from using a zip file.

Re: Mounting tar archives as a filesystem in WebAssembly

#24

Post by phiresky »

vlovich123 wrote: post #18
phiresky wrote: post #13

Earlier quoted context omitted.

I think you're looking at a different perspective than me. At _build time_ you need to process O(n), yes, and generate O(n) additional data. But I said "The amount of data you need to decompress is a constant". At _read time_, you need to do exactly three steps: 1. Load the file index - this one scales with the number of files unless you do something else smart and get it down to O(log(n)). This gives you an offset i…

My bad. Yes from decompression perspective you have O(1) ancillary data to initiate decompression at 1 seek point. This is how seeking can work in encrypted data btw without the ancillary data - you just increment the IV every N bytes so there’s a guaranteed mapping for how to derive the IV for a block so you’re bounded by how much extra you need to encrypt/decrypt to do a random byte range access of the plaintext. B…

I actually first thought this wasn't possible at all because I'm used to zstd which by default uses a 128MB window and I usually set it to the max (2GB window). 32kB is _really_ tiny in comparison. On the other hand though, zstd also compresses in parallel by default and has tools built in to handle these things, so seekable zstd archives are fairly common.

Re: Mounting tar archives as a filesystem in WebAssembly

#25

Post by kevin_thibedeau »

nrclark wrote: post #11
johannes1234321 wrote: post #6

Earlier quoted context omitted.

When looking at established file formats, I'd start with zip for that usecase over tarballs. zip has compression and ability to access any file. A tarfule you have to uncompress first. SquashFS or cramps or such have less tooling, which makes the usage for generating, inspecting, ... more complex.

You only have to decompress it first if it's compressed (commonly using gzip, which is shown with the .gz suffix). Otherwise, you can randomly access any file in a .tar as long as: - the file is seekable/range-addressible - you scan through it and build the file index first, either at runtime or in advance. Uncompressed .tar is a reasonable choice for this application because the tools to read/write tar files are ver…

Romfs is more capable, simple to support, and doesn't have the overhead of tar's large headers and typical large blocking factors.

Re: Mounting tar archives as a filesystem in WebAssembly

#26

Post by chungy »

QuantumNomad_ wrote: post #23
chungy wrote: post #20

Earlier quoted context omitted.

Zip allows you to set compression algorithm on a per-file basis, including no compression.

You can achieve the same with tar if you individually compress the files before adding them to the tar ball instead of compressing the tar ball itself. I don’t see how that plus a small index of offsets would be notably more or less work to do from using a zip file.

Zip has a central directory you could just query, instead of having to construct one in-memory by scanning the entire archive. That's significantly less work.

Re: Mounting tar archives as a filesystem in WebAssembly

#27

Post by Dwedit »

TAR archives are good in a few ways, but random access to files is not one of them. You need to iterate over every file before you can create a mapping between filename and its TAR file address.

(Meanwhile, sending TAR over Netcat is a valid way to clone a filesystem to another computer, including maintaining the hardlinks and symlinks)

Re: Mounting tar archives as a filesystem in WebAssembly

#28

Post by tredre3 »

crabique wrote: post #22

Very cool, I wish there were something similar to this for filesystem images though. Just recently I needed to somehow generate a .tar.gz from a .raw ext4 image and, surprisingly, there's still no better option than actually mounting it and then creating an archive. I managed to "isolate" it a bit with guestfish's tar-out, but still it's pretty slow as it needs to seek around the image (in my case over NBD) to get th…

There are surprisingly few tools to work on file system images in the Linux world, they expect loopback mounting to always be available.

There are a few libraries to read ext4 but every time I've tried to use one it missed one feature that my specific image was using (mke2fs changes its defaults every couple years to rely on newer ext4 features).

7-zip can also read ext4 to some degree and, I'm not sure but, they seem to have written a naive parser of their own to do it: https://github.com/mcmilk/7-Zip/blob/master/CPP/7zip/Archive...

Re: Mounting tar archives as a filesystem in WebAssembly

#30

Post by QuantumNomad_ »

chungy wrote: post #26
QuantumNomad_ wrote: post #23

Earlier quoted context omitted.

You can achieve the same with tar if you individually compress the files before adding them to the tar ball instead of compressing the tar ball itself. I don’t see how that plus a small index of offsets would be notably more or less work to do from using a zip file.

Zip has a central directory you could just query, instead of having to construct one in-memory by scanning the entire archive. That's significantly less work.

I mean if they include a pre-made index with it. For example an uncompressed index at byte offset 0 in the tar ball that lists what is inside and their offsets. It would still be comparable amount of work to create software to do that with tar as to use a zip file, if fine grained compression levels etc is being used.
Post reply on HN