Live data from Hacker News

How are zlib, gzip and Zip related? (2013)

stackoverflow.com

101–110 of 145 posts

Re: How are zlib, gzip and Zip related? (2013)

#101
post #78

Earlier quoted context omitted.

The patent is invalid if prior art exists,so publishing it is enough to avoid it being poached by others

But doesn't the patent office only look at previous patents when searching for prior art? Of course, other sources of prior art can be brought in as a defense, but it still may not stop someone from getting a patent initially and using it to harass financially weak opponents.

Lack of patent examiner familiarity with the industry, history, and the state of the art is traditionally considered a weak point in the system as a whole. Supposedly when writing the patent claims you're supposed to do a search to make sure you're not conflicting with other patents and that there is no prior art, but this is often used to word the patent in such a way to avoid the prior art from invalidating the patent, or use differences with the prior art as support for the uniqueness of the patent being applied for. Unfortunately, patent examiners are traditionally illequipped to accurately make an assessment, and the patent applicant is allowed to make adjustments to address the issues the patent examiner found. I remember there was work on getting patent examiners to be more familiar with the areas the patents they are examining/approving are for (technology, software, or business, for example), but I don't know where that stands today.

However, IANAPL and IANAPE.

Re: How are zlib, gzip and Zip related? (2013)

#102
post #94

Earlier quoted context omitted.

>> That's as close as you're gonna get to "primary source" folks! Yes it is. Whenever I think of PKzip I am reminded of the sad tale of Phil Katz death - partly because I share his first and last initials, and now reading his Wikipedia page I see we had more in common. Fortunately alcoholism is not my thing. https://en.wikipedia.org/wiki/Phil_Katz

I remember reading that the creators of the ARC format sued Phil Katz for essentially taking their original source code and rebranding it PKARC (with assembly optimizations). They claim [1] that Katz did essentially the same thing with PKZIP. Can anyone verify these claims? I always thought it was an injustice that the original authors were treated poorly by the BBS community and treated like a big company despite ju…

This is covered in the BBS Documentary[1]. A comparison of Phil Katz work showed he just renamed variables and moved things around. Phil rallied the BBS community to character assassinate Thom. The documentary is shot years later and Thom still breaks down and cries when talking about what happened. It is very sad.

[1] http://bbsdocumentary.com/

Re: How are zlib, gzip and Zip related? (2013)

#103
post #96

Earlier quoted context omitted.

I followed your link which leads me to rant a bit: LZFSE has been out since one year. Not one mention on wikipedia. The repo lacks a good description what lzfse is. It also contains a LZVN encoder/decoder. The only information I found about it is some blog where someone seems to reverse engineer it for some hackintosh purposes. I know documenting and presenting the case why people should use your software/file format…

WWDC video talking about it: https://developer.apple.com/videos/play/wwdc2015/712/ Apple's docs: https://developer.apple.com/library/mac/documentation/Perfor... Generic info on finite state entropy compression: http://fastcompression.blogspot.com/2013/12/finite-state-ent... The last link was submitted to HN: https://news.ycombinator.com/item?id=7040951 The repo's README seems pretty clear. Google provides the rest. N…

Those first two links should ideally have been included in the project's README. From a maintainer's perspective, "Google it" should never be acceptable.

Re: How are zlib, gzip and Zip related? (2013)

#105

Earlier quoted context omitted.

Out of curiosity, why do you care whether the answer is marked as accepted?

Because it more likely that beginning programmers would jump straight and look for the "accepted answer".

What mechanism would you suggest otherwise?

For example, suppose I ask a question, and within 10 minutes get an answer which is good enough for me, then I accept it and move on.

Hours later, someone gives a much better answer. What is my obligation to track the topic after I already know a correct answer? What should be the mechanism to override my acceptance?

What is more achievable: putting that mechanism in place, or getting beginning programmers to look at the first answer rather than looking for the accepted one?

To which I'll add, I'm looking at it in privacy mode (not logged in), and can't tell which is accepted. Is that something that only people with an account can see? If so, aren't those also people who are now no longer beginner SO users?

Re: How are zlib, gzip and Zip related? (2013)

#106
post #78

Earlier quoted context omitted.

But doesn't the patent office only look at previous patents when searching for prior art? Of course, other sources of prior art can be brought in as a defense, but it still may not stop someone from getting a patent initially and using it to harass financially weak opponents.

Lack of patent examiner familiarity with the industry, history, and the state of the art is traditionally considered a weak point in the system as a whole. Supposedly when writing the patent claims you're supposed to do a search to make sure you're not conflicting with other patents and that there is no prior art, but this is often used to word the patent in such a way to avoid the prior art from invalidating the pat…

http://patents.stackexchange.com/ helps with this

Re: How are zlib, gzip and Zip related? (2013)

#107
post #77
post #41

Earlier quoted context omitted.

The tar.gz format is to combine a collection of files first as a single tar file and then compress the one file into gz. It's not compressing individual file and then combing the compressed files into a tar. Accessing the last file requires decompressing the whole archive. What you described is the zip format. Both zip and gz use the DEFLATE algorithm so there's no difference in compression. The difference is in how…

> It's not compressing individual file and then combing the compressed files into a tar. Accessing the last file requires decompressing the whole archive. Yes, that is the whole point of this discussion. This limitation causes problems for some. > What you described is the zip format. That is not the zip format. Zip is a collection of file objects with uncompressed headers which specify the metadata and size of the c…

[deleted]

Re: How are zlib, gzip and Zip related? (2013)

#108
post #55

In other compression news, Apple open sourced their implementation of lzfse yesterday: https://github.com/lzfse/lzfse . It's based on a relatively new type of coding - asymmetric numeral systems. Huffman coding is only optimal if you consider one bit as the smallest unit of information. ANS (and more broadly, arithmetic coding) allows for fractional bits and gets closer to the Shannon limit. It's also simpler to impl…

Apple tANS implementation was suboptimal - both from the quantization and symbol spread point of view: http://encode.ru/threads/2221-LZFSE-New-Apple-Data-Compressi... The author of this post (Eric Biggers) has made corrections and one of them was already merged in lzfse github (the second would make it incompatible).

There is a paper about FPGA implementation of tANS encoder ( ieeexplore.ieee.org/xpl/login.jsp?tp=&arnumber=7306068 ), but it uses branch which is removed in more recent implementations (bottom of the first post of http://encode.ru/threads/2078-List-of-Asymmetric-Numeral-Sys... ).

It is worth to mention that rANS variant - using one multiplication per symbol (e.g. in Google VP10), has recently exceeded the speed of tANS/FSE: https://github.com/jkbonfield/rans_static

Re: How are zlib, gzip and Zip related? (2013)

#109
post #102
post #94

Earlier quoted context omitted.

I remember reading that the creators of the ARC format sued Phil Katz for essentially taking their original source code and rebranding it PKARC (with assembly optimizations). They claim [1] that Katz did essentially the same thing with PKZIP. Can anyone verify these claims? I always thought it was an injustice that the original authors were treated poorly by the BBS community and treated like a big company despite ju…

This is covered in the BBS Documentary[1]. A comparison of Phil Katz work showed he just renamed variables and moved things around. Phil rallied the BBS community to character assassinate Thom. The documentary is shot years later and Thom still breaks down and cries when talking about what happened. It is very sad. [1] http://bbsdocumentary.com/

Crazy how that documentary appears to be not for sale in any form and DVDs on Amazon go for $3xx. Am I missing something?

Re: How are zlib, gzip and Zip related? (2013)

#110
post #77
post #41

Earlier quoted context omitted.

The tar.gz format is to combine a collection of files first as a single tar file and then compress the one file into gz. It's not compressing individual file and then combing the compressed files into a tar. Accessing the last file requires decompressing the whole archive. What you described is the zip format. Both zip and gz use the DEFLATE algorithm so there's no difference in compression. The difference is in how…

> It's not compressing individual file and then combing the compressed files into a tar. Accessing the last file requires decompressing the whole archive. Yes, that is the whole point of this discussion. This limitation causes problems for some. > What you described is the zip format. That is not the zip format. Zip is a collection of file objects with uncompressed headers which specify the metadata and size of the c…

> The stream format works in blocks of up to 65k.

Can you point to where in the gzip or DEFLATE spec having that stipulation?

The gzip format doesn't store the compressed data length, nor the offset of the compressed data blobs. It just has the header and then the blobs of DEFLATE compressed data, one after another. That's why it's good for streaming.

Same thing with DEFLATE. It's just a series of arbitrary size compressed data blocks. With a dictionary upfront for dynamic Huffman and no dictionary for static Huffman. You read as much data as you can to decompress until encountering the ending Huffman symbol, where you arrive at the next block. The 65K limit is for non-compress type block, which is not useful in the scheme of things.

There's no well known magic signature bytes to search for a DEFLATE header.

Post reply on HN