Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Ziplm: Gzip-Backed Language Model

github.com

Post reply on HN
Print view View original

51–59 of 59 posts

Re: Ziplm: Gzip-Backed Language Model

#51

Post by canjobear »

userbinator wrote: post #13
canjobear wrote: post #2

Oh crud, I made the front page. It should be obvious that this is junk quality as a language model. But it's a cool example of the equivalence between compression codes and probability distributions, so I hope people find it interesting for that reason. You can also get a bit of an intuitive sense for the patterns that gzip and bzip2 pick up on in text (they like repetitive strings).

Repeating this experiment with a more optimal compression algorithm might be interesting too.

It doesn't look qualitatively any better with bzip2 or lzma.

Re: Ziplm: Gzip-Backed Language Model

#52

Post by jdiff »

pseg134 wrote: post #46
optimalsolver wrote: post #38

Earlier quoted context omitted.

Strong disagree. In AI, many cool ideas blow up when meeting anything close to real world inputs.

That is the perfect attitude for you then. Please ignore this toy and use only the Webscale tech blessed by google.

This is a horribly disingenuous and dismissive response. Sometimes good ideas fundamentally just don't work, they miss foundational issues that make them incompatible with reality. That just happens sometimes. Not everything can be polished up and made into something useful.

Re: Ziplm: Gzip-Backed Language Model

#53

Post by canjobear »

vintermann wrote: post #30
userbinator wrote: post #13

Earlier quoted context omitted.

Repeating this experiment with a more optimal compression algorithm might be interesting too.

If I understand it correctly, this implementation re-compresses the prefix 256 times, once for each possible byte continuation. It should be possible to save a lot of work by saving the state of the compressor just before the predicted byte, and reusing it. Also, since gzip works on a limited sliding context window, it's probably pointless to give it a prefix much longer than the window... unless there's something ab…

I'd be happy to work on improving this with someone who understands the internals of the compressors.

Re: Ziplm: Gzip-Backed Language Model

#54

Post by jll29 »

This is a nice application of the idea that understanding and compression are equivalent, which is the inspiration behind the Hutter prize (500 K€ for whoever can compress Wikipedia best, see http://prize.hutter1.net/ - the current state of the art is about 15 MB).

Compression can also be used as "machine learning method": but all data vectors with the same class label into a separate bucket; any new, unseen and unlabeled data item can be added to each bucket. The most likely class is the one the bucket of which grows the least when compressed after adding it. The University of Waikato group (Ian Witten and co-workers) did a fair amount of that kind of work, perhaps first, e.g. Frank, Eibe, Chang Chui and Ian H. Witten (2000) "Text categorization using compression models", https://www.cs.waikato.ac.nz/~eibe/pubs/Frank_categorization... - yes, published 23 years ago!).

Re: Ziplm: Gzip-Backed Language Model

#55

Post by Pannoniae »

jll29 wrote: post #54

This is a nice application of the idea that understanding and compression are equivalent, which is the inspiration behind the Hutter prize (500 K€ for whoever can compress Wikipedia best, see http://prize.hutter1.net/ - the current state of the art is about 15 MB). Compression can also be used as "machine learning method": but all data vectors with the same class label into a separate bucket; any new, unseen and unla…

(small correction: the current record is 115MB, not 15MB)

Re: Ziplm: Gzip-Backed Language Model

#56

Post by puttycat »

For anyone interested in the idea of compression as learning combined with neural nets, a random list of interesting work:

1. Schmidhuber's classical work applying a time-based Kolmogorov complexity to neural nets

https://pubmed.ncbi.nlm.nih.gov/12662875/

2. Applying a form of Kolmogorov complexity to learn formal languages using RNNs

https://direct.mit.edu/tacl/article/doi/10.1162/tacl_a_00489...

3. Hinton and Van Camp - Keeping the neural networks simple by minimizing the description length of the weights

https://dl.acm.org/doi/10.1145/168304.168306

Re: Ziplm: Gzip-Backed Language Model

#57

Post by nunesvn »

Pannoniae wrote: post #55
jll29 wrote: post #54

This is a nice application of the idea that understanding and compression are equivalent, which is the inspiration behind the Hutter prize (500 K€ for whoever can compress Wikipedia best, see http://prize.hutter1.net/ - the current state of the art is about 15 MB). Compression can also be used as "machine learning method": but all data vectors with the same class label into a separate bucket; any new, unseen and unla…

(small correction: the current record is 115MB, not 15MB)

(small clarification: it’s not the complete wikipedia, just a 1GB slice of it)

Re: Ziplm: Gzip-Backed Language Model

#58

Post by up2isomorphism »

dleeftink wrote: post #4
canjobear wrote: post #2

Oh crud, I made the front page. It should be obvious that this is junk quality as a language model. But it's a cool example of the equivalence between compression codes and probability distributions, so I hope people find it interesting for that reason. You can also get a bit of an intuitive sense for the patterns that gzip and bzip2 pick up on in text (they like repetitive strings).

I think it is a cool 'tangible' demo of how non-human readable data remains learnable for ML purposes. I've recently tried using Nilsimsa and sdeep hashes to reduce the input dimensionality of natural text, with some loss in accuracy on similarity tasks. A combination of byte-pair encoding and local-sensitive hashing might prove a more stable combination. Throw some gzip function that way and we may be able to reduce…

Why does this needs to be "demoed"? The weights for ML inference is obviously "binary".

Re: Ziplm: Gzip-Backed Language Model

#59

Post by gr0m »

canjobear wrote: post #2

Oh crud, I made the front page. It should be obvious that this is junk quality as a language model. But it's a cool example of the equivalence between compression codes and probability distributions, so I hope people find it interesting for that reason. You can also get a bit of an intuitive sense for the patterns that gzip and bzip2 pick up on in text (they like repetitive strings).

Have you tried beam search as well?
Post reply on HN