It's kinda bad-ass when he said: you can use this text on Wikipedia, I'm the primary reference.
Wikipedia mostly doesn't allow primary sources though. It needs to be corroborated by a reliable secondary source.
How are zlib, gzip and Zip related? (2013)
31–40 of 145 posts
Re: How are zlib, gzip and Zip related? (2013)
#32It's kinda bad-ass when he said: you can use this text on Wikipedia, I'm the primary reference.
I wonder if that's a bit of snark about how, if he were to actually edit the Wikipedia entry on the topic, his edits would be removed because of Wikipedia's policies on primary sources contributing to an entry.
Re: How are zlib, gzip and Zip related? (2013)
#33When I read "I am the reference" it reminded me of "I am the danger".
Re: How are zlib, gzip and Zip related? (2013)
#34One important difference in practice is that zip files needs to be saved to disk to be extracted. gzip files on the other hand can be stream unzipped i.e curl http://example.com/foo.tar.gz | tar zxvf - is possible but not with zip files. I am not sure if this is a limitation of the unzip tool. I would love to know if there is a work around to this.
You can't do that with a .zip file because the file header information is actually at the end of the file. You could work around that by sending the header first, though.
To clarify: if it's HTTP, you can request the end of file "header" first if the server supports the `Range` HTTP header (many servers support this when it's appropriate). So it's not doable by simple bash-style piping, but a dedicated tool can request the required bytes in the correct order.
I'm pretty sure I've used something similar, which was already implemented in the .NET framework. See their docs on constructing a ZipArchive object:
"If the underlying file or stream supports seeking, the files are read from the archive as they are requested. If the underlying file or stream does not support seeking, the entire archive is held in memory."
https://msdn.microsoft.com/en-us/library/system.io.compressi...
Re: How are zlib, gzip and Zip related? (2013)
#35One important difference in practice is that zip files needs to be saved to disk to be extracted. gzip files on the other hand can be stream unzipped i.e curl http://example.com/foo.tar.gz | tar zxvf - is possible but not with zip files. I am not sure if this is a limitation of the unzip tool. I would love to know if there is a work around to this.
Zip files can be streamed. See for example https://developer.android.com/reference/java/util/zip/ZipInp...
It even allows you to stream the zip while creating it, i.e. you do not need to know the compressed size of the files before starting to write the compressed file content.
Re: How are zlib, gzip and Zip related? (2013)
#36Earlier quoted context omitted.
If I recall correctly and it's been a while: The initial header is x bytes from the beginning of the file, or that you have to search for a known key string PKsomething. Then you have to go back and forth between that and the compressed data as you decompress. When compressing files you have to go back to the beginning of the zip file ( or disk1 in a multi disk archive) and update that table with the CRC info. Just w…
That can't be the case anymore because I implemented a streaming function a few years back that sends an unlimited number of files as a zip, but builds the zip on the fly streaming it to the client. Maybe it works because I am not using compression (files in my case are all JPG and/or compressed video so there is little benefit). But my process definitely starts streaming the zip right away and has to pull URLs on th…
I'm pretty sure you were using the deflate stream format. There may be other formats (maybe with different magic numbers in the header) that have slightly different capabilities and needs.
Re: How are zlib, gzip and Zip related? (2013)
#37Earlier quoted context omitted.
You can't do that with a .zip file because the file header information is actually at the end of the file. You could work around that by sending the header first, though.
How do you send the header first? Is that a feature of the http framework?
It'd be something you'd have to write yourself, of course.
On the server side, read it and send it before the file. Maybe go through all your zips and copy the needed metadata to another location.
Re: How are zlib, gzip and Zip related? (2013)
#38Earlier quoted context omitted.
You can't do that with a .zip file because the file header information is actually at the end of the file. You could work around that by sending the header first, though.
You can't put the header first. It is an intentional feature of .zip that the header is at the end so you can update a large zip file by just appending a new header to the end. That way the entire file does not have to be rewritten. Just read the old header, append new files, append new header. This was important back in floppy disk days
>> Just read the old header, append new files, append new header. This was important back in floppy disk days
Don't forget 'overwrite old header'
Important for floppy disks in two ways, one because of space constraints and two, because of how slow floppies were.
Re: How are zlib, gzip and Zip related? (2013)
#39Not only is this a great read, but the follow up for citations is replied with "I am the reference". If this were reddit I'd post the hot fire gif. Eh, here it's anyway: http://i.imgur.com/VQLGJOL.gif
I thought he was joking. Then I read the name: Adler: as in Adler-32 (the checksum function that zlib uses). Then I knew it was real. The darn author of zlib answered the question. That's as close as you're gonna get to "primary source" folks!
Mark Adler is such a boss. Check out his answers to space questions on http://space.stackexchange.com/users/265/mark-adler
Re: How are zlib, gzip and Zip related? (2013)
#40Not only is this a great read, but the follow up for citations is replied with "I am the reference". If this were reddit I'd post the hot fire gif. Eh, here it's anyway: http://i.imgur.com/VQLGJOL.gif
Neat how involved he still is this long after the initial work.
When brotli came out he implemented his own decoder (https://github.com/madler/brotli/) to check the spec, and got some things clarified in RFC. Brotli is pretty DEFLATE-y in spirit, with changes that take advantage of today's hardware (larger history window, contexts in the entropy encoding) and a bunch of tweaks (different match encoding, static dictionary, etc.).
Also, as others mention, NASA mission engineer: https://en.wikipedia.org/wiki/Mark_Adler#Mars_exploration