Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

How are zlib, gzip and Zip related? (2013)

stackoverflow.com

Post reply on HN
Print view View original

41–50 of 145 posts

Re: How are zlib, gzip and Zip related? (2013)

#41

Post by ww520 »

kbenson wrote: post #17

It seems like it wouldn't be that hard to create an indexed tar.gz format that's backwards compatible. One way would be to use the last file in the tar as the index, and as files are added, you can remove the index, append the new file, append some basic file metadata and the compressed offset (maybe of the deflate chunk) into the index, update the index size in bytes in a small footer at the end of the index, and ap…

The tar.gz format is to combine a collection of files first as a single tar file and then compress the one file into gz. It's not compressing individual file and then combing the compressed files into a tar. Accessing the last file requires decompressing the whole archive. What you described is the zip format. Both zip and gz use the DEFLATE algorithm so there's no difference in compression. The difference is in how the files are packaged.

Also gzip supports on-the-fly stream-based compression and decompression so anything requires jumping to the end to do reading and writing while compression/decompresion is no go.

Re: How are zlib, gzip and Zip related? (2013)

#44

Post by [deleted] »

tobias3 wrote: post #35
the_common_man wrote: post #14

One important difference in practice is that zip files needs to be saved to disk to be extracted. gzip files on the other hand can be stream unzipped i.e curl http://example.com/foo.tar.gz | tar zxvf - is possible but not with zip files. I am not sure if this is a limitation of the unzip tool. I would love to know if there is a work around to this.

Zip files can be streamed. See for example https://developer.android.com/reference/java/util/zip/ZipInp... It even allows you to stream the zip while creating it, i.e. you do not need to know the compressed size of the files before starting to write the compressed file content.

[deleted]

Re: How are zlib, gzip and Zip related? (2013)

#45

Post by pierrec »

tobias3 wrote: post #35
the_common_man wrote: post #14

One important difference in practice is that zip files needs to be saved to disk to be extracted. gzip files on the other hand can be stream unzipped i.e curl http://example.com/foo.tar.gz | tar zxvf - is possible but not with zip files. I am not sure if this is a limitation of the unzip tool. I would love to know if there is a work around to this.

Zip files can be streamed. See for example https://developer.android.com/reference/java/util/zip/ZipInp... It even allows you to stream the zip while creating it, i.e. you do not need to know the compressed size of the files before starting to write the compressed file content.

I wonder how their API degrades when the stream doesn't support seeking (as is often the case over HTTP, or like in gp's example of using pipes). The whole doc you linked to, and the feature of on-the-fly decompression, seems to assume that the stream is seekable.

Re: How are zlib, gzip and Zip related? (2013)

#46

Post by joosters »

bluedino wrote: post #38
greggman wrote: post #28

Earlier quoted context omitted.

You can't put the header first. It is an intentional feature of .zip that the header is at the end so you can update a large zip file by just appending a new header to the end. That way the entire file does not have to be rewritten. Just read the old header, append new files, append new header. This was important back in floppy disk days

>> Just read the old header, append new files, append new header. This was important back in floppy disk days Don't forget 'overwrite old header' Important for floppy disks in two ways, one because of space constraints and two, because of how slow floppies were.

Don't forget 'overwrite old header'

Why? The new one will become the 'real' one since it will be at the end of the modified file. So it doesn't really matter if you delete the old one. If you're really trying to squeeze file sizes down, you could reference the old 'header' from the new one, so that you do not have to list the entire archive's contents again.

Re: How are zlib, gzip and Zip related? (2013)

#48

Post by tobias3 »

pierrec wrote: post #45
tobias3 wrote: post #35

Earlier quoted context omitted.

Zip files can be streamed. See for example https://developer.android.com/reference/java/util/zip/ZipInp... It even allows you to stream the zip while creating it, i.e. you do not need to know the compressed size of the files before starting to write the compressed file content.

I wonder how their API degrades when the stream doesn't support seeking (as is often the case over HTTP, or like in gp's example of using pipes). The whole doc you linked to, and the feature of on-the-fly decompression, seems to assume that the stream is seekable.

InputStreams are not seekable (see https://developer.android.com/reference/java/io/InputStream.... ).

You can also have a look at a zip file spec to see that they can be streamed: https://users.cs.jmu.edu/buchhofp/forensics/formats/pkzip.ht...

Re: How are zlib, gzip and Zip related? (2013)

#50

Post by brianwawok »

tdicola wrote: post #42

Wow a stackoverflow question that hasn't been closed or removed for some trivial reason--thought I'd never see something like that again.

You think stackoverflow has too many good questions being closed? If anything I think more could be closed.
Post reply on HN