Archived link juste in case http://archive.is/SvUO5
How are zlib, gzip and Zip related? (2013)
11–20 of 145 posts
Re: How are zlib, gzip and Zip related? (2013)
#12Re: How are zlib, gzip and Zip related? (2013)
#13It's annoyingly common how the OP doesn't mark this answer as accepted, or even acknowledge how amazing this answer is from one of the technology's creators -- instead just goes on to ask a followup.
The accepted answer is often wrong (or obsolete) and grants a small number of points. The obsession with it on SO is a bit odd.
Yeah I tend to think if another answer has double the points of an accepted answer, that answer should come first.
Re: How are zlib, gzip and Zip related? (2013)
#14One important difference in practice is that zip files needs to be saved to disk to be extracted. gzip files on the other hand can be stream unzipped i.e curl http://example.com/foo.tar.gz | tar zxvf - is possible but not with zip files. I am not sure if this is a limitation of the unzip tool. I would love to know if there is a work around to this.
Re: How are zlib, gzip and Zip related? (2013)
#15Archived link juste in case http://archive.is/SvUO5
The post is over two years old and has a score of +892 (easily in the top 0.1% of posts). It's not going anywhere.
Re: How are zlib, gzip and Zip related? (2013)
#16One important difference in practice is that zip files needs to be saved to disk to be extracted. gzip files on the other hand can be stream unzipped i.e curl http://example.com/foo.tar.gz | tar zxvf - is possible but not with zip files. I am not sure if this is a limitation of the unzip tool. I would love to know if there is a work around to this.
You can't do that with a .zip file because the file header information is actually at the end of the file. You could work around that by sending the header first, though.
Re: How are zlib, gzip and Zip related? (2013)
#17It seems like it wouldn't be that hard to create an indexed tar.gz format that's backwards compatible.
One way would be to use the last file in the tar as the index, and as files are added, you can remove the index, append the new file, append some basic file metadata and the compressed offset (maybe of the deflate chunk) into the index, update the index size in bytes in a small footer at the end of the index, and append to the compressed tar (add).
You can retrieve the index by starting at the end of the compressed archive, and reading backwards until you find a deflate header (at most 65k plus a few more bytes, since that's the size of a deflate chunk), If it's an indexed tar, the last file will be the index, and the end of the index will be a footer with the index size (so you know the maximum you'll need to seek back from the end). This isn't extremely efficient, but it is limited in scope, and helped by knowing the index size.
You could verify the index by checking some or all of the reported file byte offsets. Worst case scenario is small files with one or more per deflate chunk, and you would have to visit each chunk. This makes the worst case scenario equivalent to listing files an un-indexed tar.gz, plus the overhead of locating and reading the index (relatively small).
Uncompressing the archive as a regular tar.gz would result in a normal operation, with an additional file (the index) included.
I imagine this isn't popular is not because it hasn't been done, but because most people don't really need an index.
Re: How are zlib, gzip and Zip related? (2013)
#18It seems like it wouldn't be that hard to create an indexed tar.gz format that's backwards compatible. One way would be to use the last file in the tar as the index, and as files are added, you can remove the index, append the new file, append some basic file metadata and the compressed offset (maybe of the deflate chunk) into the index, update the index size in bytes in a small footer at the end of the index, and ap…
After writing one myself along with a guy on IRC who needed a 'quick way to access huge .tar files', I found one written in python: https://github.com/devsnd/tarindexer
He ended up doing the smart thing and changing his archiving program to not use .tar files, which solved all of his problems, but for the 2TB of data he already had this worked rather well.
Re: How are zlib, gzip and Zip related? (2013)
#19It seems like it wouldn't be that hard to create an indexed tar.gz format that's backwards compatible. One way would be to use the last file in the tar as the index, and as files are added, you can remove the index, append the new file, append some basic file metadata and the compressed offset (maybe of the deflate chunk) into the index, update the index size in bytes in a small footer at the end of the index, and ap…
If I recall correctly and it's been a while: The initial header is x bytes from the beginning of the file, or that you have to search for a known key string PKsomething. Then you have to go back and forth between that and the compressed data as you decompress. When compressing files you have to go back to the beginning of the zip file ( or disk1 in a multi disk archive) and update that table with the CRC info. Just was not practical for streaming for multiple reasons. Remember the original zip file format was created by phil in about 1987. I believe they just thought it easier to start over with a better design for gzip with the same compression algorithm.
Re: How are zlib, gzip and Zip related? (2013)
#20It seems like it wouldn't be that hard to create an indexed tar.gz format that's backwards compatible. One way would be to use the last file in the tar as the index, and as files are added, you can remove the index, append the new file, append some basic file metadata and the compressed offset (maybe of the deflate chunk) into the index, update the index size in bytes in a small footer at the end of the index, and ap…
After writing one myself along with a guy on IRC who needed a 'quick way to access huge .tar files', I found one written in python: https://github.com/devsnd/tarindexer He ended up doing the smart thing and changing his archiving program to not use .tar files, which solved all of his problems, but for the 2TB of data he already had this worked rather well.
Yeah, I almost wrote one myself in Perl a year or two ago. :)
I was going for maximal tar compatibility, graceful fallback, low impact file additions, and the ability to add an index after the tar is created without rewriting it all. That's what I came up with off the top of my head.
I did just search for indexed tar though, and it came up with dar[1], which I had heard of, but completely forgotten about.