Isn't "archive" embedded in "tar" already? In other words, is this like saying one went to the "ATM machine"?
Redundancy in natural language isn't a big deal, and it isn't entirely useless, either.
Mounting tar archives as a filesystem in WebAssembly
31–40 of 42 posts
Re: Mounting tar archives as a filesystem in WebAssembly
#32I'm a bit disappointed that this only solves the "find index of file in tar" problem, but not at all the "partially read a tar.gz" file problem. So really you're still reading the whole file into memory, so why not just extract the files properly while you are doing that? Takes the same amount of time (O(n)) and less memory. The gzip-random-access problem one is a lot more difficult because the gzip has internal stat…
If anyone knows a similar solution for zstd, I'm very interested. I'm doing streaming uncompression to disk and I'd like to be able to do resumable downloads without _also_ storing the compressed file.
Re: Mounting tar archives as a filesystem in WebAssembly
#33I'm a bit disappointed that this only solves the "find index of file in tar" problem, but not at all the "partially read a tar.gz" file problem. So really you're still reading the whole file into memory, so why not just extract the files properly while you are doing that? Takes the same amount of time (O(n)) and less memory. The gzip-random-access problem one is a lot more difficult because the gzip has internal stat…
The first time I'd heard of this was via https://github.com/jonjohnsonjr/dagdotdev/blob/main/internal... which powers https://oci.dag.dev to let you browse OCI images (e.g., https://oci.dag.dev/fs/ubuntu@sha256:b40150c1c2717d324cdb172... )
This is very cool. Worth a submission by itself.
Re: Mounting tar archives as a filesystem in WebAssembly
#34Isn't "archive" embedded in "tar" already? In other words, is this like saying one went to the "ATM machine"?
Redundancy in natural language isn't a big deal, and it isn't entirely useless, either.
Ok, insofar as saying "tape archive archive" out loud doesn't sound odd to you, keep doing it I suppose.
Re: Mounting tar archives as a filesystem in WebAssembly
#35I'm a bit disappointed that this only solves the "find index of file in tar" problem, but not at all the "partially read a tar.gz" file problem. So really you're still reading the whole file into memory, so why not just extract the files properly while you are doing that? Takes the same amount of time (O(n)) and less memory. The gzip-random-access problem one is a lot more difficult because the gzip has internal stat…
If anyone knows a similar solution for zstd, I'm very interested. I'm doing streaming uncompression to disk and I'd like to be able to do resumable downloads without _also_ storing the compressed file.
https://github.com/martinellimarco/indexed_zstd
https://github.com/martinellimarco/libzstd-seek
Note, however, that this can only seek to frames, and zstd still only creates files containing a single frame by default. pzstd did create multi-frame files, but it is not being developed anymore. Other alternatives for creating seekable zstd files are: zeekstd, t2sz, and zstd-seekable-format-go.
Re: Mounting tar archives as a filesystem in WebAssembly
#36Earlier quoted context omitted.
If anyone knows a similar solution for zstd, I'm very interested. I'm doing streaming uncompression to disk and I'd like to be able to do resumable downloads without _also_ storing the compressed file.
https://github.com/martinellimarco/indexed_zstd https://github.com/martinellimarco/libzstd-seek Note, however, that this can only seek to frames, and zstd still only creates files containing a single frame by default. pzstd did create multi-frame files, but it is not being developed anymore. Other alternatives for creating seekable zstd files are: zeekstd, t2sz, and zstd-seekable-format-go.
Thanks, this is helpful. I might just end up using content defined chunking in addition/instead, but it's good to know that there is a path forward if I stick with the current architecture.
Re: Mounting tar archives as a filesystem in WebAssembly
#37Earlier quoted context omitted.
Zip has a central directory you could just query, instead of having to construct one in-memory by scanning the entire archive. That's significantly less work.
I mean if they include a pre-made index with it. For example an uncompressed index at byte offset 0 in the tar ball that lists what is inside and their offsets. It would still be comparable amount of work to create software to do that with tar as to use a zip file, if fine grained compression levels etc is being used.
But then you are not using tar, you are doing your own file format atop of tar.
Re: Mounting tar archives as a filesystem in WebAssembly
#38Earlier quoted context omitted.
I mean if they include a pre-made index with it. For example an uncompressed index at byte offset 0 in the tar ball that lists what is inside and their offsets. It would still be comparable amount of work to create software to do that with tar as to use a zip file, if fine grained compression levels etc is being used.
But then you are not using tar, you are doing your own file format atop of tar.
I suppose you are right about that. But it would still be a valid tar file that can be viewed and extracted with normal tools. Kind of similar to how a .docx file can be extracted as zip but still has additional structure to its contents.
Re: Mounting tar archives as a filesystem in WebAssembly
#39I'm a bit disappointed that this only solves the "find index of file in tar" problem, but not at all the "partially read a tar.gz" file problem. So really you're still reading the whole file into memory, so why not just extract the files properly while you are doing that? Takes the same amount of time (O(n)) and less memory. The gzip-random-access problem one is a lot more difficult because the gzip has internal stat…
Tar doesn't need to imply gzip (or bzip2, or zstd, etc). Tar's default operation produces uncompressed archives.
Re: Mounting tar archives as a filesystem in WebAssembly
#40[dead]