But can he invert a binary tree and is he willing to relocate to San Francisco?
We are going to need that for our middle out compression algorithm.
How are zlib, gzip and Zip related? (2013)
91–100 of 145 posts
Re: How are zlib, gzip and Zip related? (2013)
#92But can he invert a binary tree and is he willing to relocate to San Francisco?
We detached this subthread from https://news.ycombinator.com/item?id=11924944 and marked it off-topic.
Re: How are zlib, gzip and Zip related? (2013)
#93It is rare to be able to have a question answered so completely and from such a first-hand source. This post is gold and tickles me in all the right places.
StackOverflow is sitting on a veritable treasure trove of knowledge.
Re: How are zlib, gzip and Zip related? (2013)
#94Earlier quoted context omitted.
I thought he was joking. Then I read the name: Adler: as in Adler-32 (the checksum function that zlib uses). Then I knew it was real. The darn author of zlib answered the question. That's as close as you're gonna get to "primary source" folks!
>> That's as close as you're gonna get to "primary source" folks! Yes it is. Whenever I think of PKzip I am reminded of the sad tale of Phil Katz death - partly because I share his first and last initials, and now reading his Wikipedia page I see we had more in common. Fortunately alcoholism is not my thing. https://en.wikipedia.org/wiki/Phil_Katz
I remember reading that the creators of the ARC format sued Phil Katz for essentially taking their original source code and rebranding it PKARC (with assembly optimizations). They claim [1] that Katz did essentially the same thing with PKZIP.
Can anyone verify these claims? I always thought it was an injustice that the original authors were treated poorly by the BBS community and treated like a big company despite just being run from home.
Re: How are zlib, gzip and Zip related? (2013)
#95Wow a stackoverflow question that hasn't been closed or removed for some trivial reason--thought I'd never see something like that again.
You think stackoverflow has too many good questions being closed? If anything I think more could be closed.
Possibly more bad questions could be closed but more importantly SO could learn to stop trying to close good questions because of technicalities.
I guess this would take a massive effort and would have to include a rewrite of the rules if SO.
Re: How are zlib, gzip and Zip related? (2013)
#96In other compression news, Apple open sourced their implementation of lzfse yesterday: https://github.com/lzfse/lzfse . It's based on a relatively new type of coding - asymmetric numeral systems. Huffman coding is only optimal if you consider one bit as the smallest unit of information. ANS (and more broadly, arithmetic coding) allows for fractional bits and gets closer to the Shannon limit. It's also simpler to impl…
I followed your link which leads me to rant a bit: LZFSE has been out since one year. Not one mention on wikipedia. The repo lacks a good description what lzfse is. It also contains a LZVN encoder/decoder. The only information I found about it is some blog where someone seems to reverse engineer it for some hackintosh purposes. I know documenting and presenting the case why people should use your software/file format…
WWDC video talking about it:
https://developer.apple.com/videos/play/wwdc2015/712/
Apple's docs:
https://developer.apple.com/library/mac/documentation/Perfor...
Generic info on finite state entropy compression:
http://fastcompression.blogspot.com/2013/12/finite-state-ent...
The last link was submitted to HN:
https://news.ycombinator.com/item?id=7040951
The repo's README seems pretty clear. Google provides the rest. Not really sure how much more there is to say about a reference compression algorithm.
Re: How are zlib, gzip and Zip related? (2013)
#97Archived link juste in case http://archive.is/SvUO5
I appreciate your effort and sentiment, but it would be nice if the awful eyesores of ads were not plastered all over the page.
Re: How are zlib, gzip and Zip related? (2013)
#98Earlier quoted context omitted.
That can't be the case anymore because I implemented a streaming function a few years back that sends an unlimited number of files as a zip, but builds the zip on the fly streaming it to the client. Maybe it works because I am not using compression (files in my case are all JPG and/or compressed video so there is little benefit). But my process definitely starts streaming the zip right away and has to pull URLs on th…
This cannot be a 'true' zip file. The central directory is at the end of the zip file. It's made up of information that is needed for the decompression. So if you are streaming it, you cannot decompress until the entire file is reassembled.
You can definitely compose a zip file "on the fly" and stream it out. The central directory at the end of the zip file can be determined from all the content already streamed.
The only wrinkle is that each entry has a header which typically states the compressed size and checksum. Either you have to compress each entry content in some temp buffer or file to figure out the compressed size and checksum, then write out the header and compressed content. Or you write out the header with these fields zeroed, then compress the content on the fly and write it out, then write out a data descriptor with the compressed size and checksum.
Re: How are zlib, gzip and Zip related? (2013)
#99In other compression news, Apple open sourced their implementation of lzfse yesterday: https://github.com/lzfse/lzfse . It's based on a relatively new type of coding - asymmetric numeral systems. Huffman coding is only optimal if you consider one bit as the smallest unit of information. ANS (and more broadly, arithmetic coding) allows for fractional bits and gets closer to the Shannon limit. It's also simpler to impl…
More compression news!: https://github.com/Cyan4973/zstd declared its format stable a couple days ago, getting a step towards 1.0. In the example there, it's a little denser than gzip -1 and a lot faster, and includes entropy encoding that uses finite state machines (and that I don't truly understand--maybe there's some similarity to lzfse there). It can look at your data and construct a static dictionary, which sounds cool for tasks like packing a bunch of 4kb database pages.
(https://github.com/google/gipfeli is another compressor aiming at that general space ('fast but not Snappy/LZ4 fast'), but I don't think it caught on much.)
Re: How are zlib, gzip and Zip related? (2013)
#100It's annoyingly common how the OP doesn't mark this answer as accepted, or even acknowledge how amazing this answer is from one of the technology's creators -- instead just goes on to ask a followup.
Out of curiosity, why do you care whether the answer is marked as accepted?
Because it more likely that beginning programmers would jump straight and look for the "accepted answer".