Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Ziplm: Gzip-Backed Language Model

github.com

Post reply on HN
Print view View original

11–20 of 59 posts

Re: Ziplm: Gzip-Backed Language Model

#11

Post by canjobear »

hedgehog wrote: post #10
canjobear wrote: post #2

Oh crud, I made the front page. It should be obvious that this is junk quality as a language model. But it's a cool example of the equivalence between compression codes and probability distributions, so I hope people find it interesting for that reason. You can also get a bit of an intuitive sense for the patterns that gzip and bzip2 pick up on in text (they like repetitive strings).

Maybe this was the inspiration for your project but I just saw a paper that applied compressors as part of a text classification system getting accuracy competitive with BERT. 14 lines of Python. https://aclanthology.org/2023.findings-acl.426/

Yeah, that paper made me wonder how well gzip would work as a language model so I typed up this thing in a coffee shop

Re: Ziplm: Gzip-Backed Language Model

#12

Post by p-e-w »

It should be noted that Moby Dick is (in)famous for its very large and unusual vocabulary. Some lighter reading might yield better results for demonstration purposes. Also, while fairly long for a single novel, the number of words is still minuscule compared to what other LMs are trained on. Using the entire Gutenberg library, or a Wikipedia dump, could improve the quality dramatically.

Of course, it doesn't really matter, as the whole thing is obviously just a toy, but I still think that this approach should be able to produce much better output than the garbled Moby Dick example.

Re: Ziplm: Gzip-Backed Language Model

#13

Post by userbinator »

canjobear wrote: post #2

Oh crud, I made the front page. It should be obvious that this is junk quality as a language model. But it's a cool example of the equivalence between compression codes and probability distributions, so I hope people find it interesting for that reason. You can also get a bit of an intuitive sense for the patterns that gzip and bzip2 pick up on in text (they like repetitive strings).

Repeating this experiment with a more optimal compression algorithm might be interesting too.

Re: Ziplm: Gzip-Backed Language Model

#14

Post by nerdponx »

userbinator wrote: post #13
canjobear wrote: post #2

Oh crud, I made the front page. It should be obvious that this is junk quality as a language model. But it's a cool example of the equivalence between compression codes and probability distributions, so I hope people find it interesting for that reason. You can also get a bit of an intuitive sense for the patterns that gzip and bzip2 pick up on in text (they like repetitive strings).

Repeating this experiment with a more optimal compression algorithm might be interesting too.

In some sense, the feature space learned by a NN is a compression algorithm.

Re: Ziplm: Gzip-Backed Language Model

#15

Post by usgroup »

Hey author, would you mind explaining the conversion from compression length to probability please? Namely this line:

scipy.special.log_softmax(-code_lengthsself.conversion(1/temperature))

Your codes are K-ary but this doesn't look like its taken into account ala the README. What is the log(256) conversion factor? What is 1/temperature for?

Re: Ziplm: Gzip-Backed Language Model

#16

Post by usgroup »

canjobear wrote: post #9
DarmokJalad1701 wrote: post #6

Earlier quoted context omitted.

I am not sure if it was intentional, but in your "Moby Dick" example, the "alphabet" vocab is missing the letter 'f'.

Ha, yeah. I was just running my finger along the keyboard to type out the string. I just regenerated it, btw, and got a better looking result.

i suggest you get all the unique characters from the moby dick text and use it as the alphabet if you're generating from it, or truncate the text to just your hand selected characters else gzip cannot even approximate an optimal code if the alphabets don't match.

Re: Ziplm: Gzip-Backed Language Model

#17

Post by fnordpiglet »

canjobear wrote: post #2

Oh crud, I made the front page. It should be obvious that this is junk quality as a language model. But it's a cool example of the equivalence between compression codes and probability distributions, so I hope people find it interesting for that reason. You can also get a bit of an intuitive sense for the patterns that gzip and bzip2 pick up on in text (they like repetitive strings).

I think this would work better if you used a fixed compression dictionary trained over the corpus and used to compress the prompt. Unless you do that? I wouldn’t expect this to work well for such a small training set. But I do wonder if this is an excellent way to improve token resolution; especially in poorly represented languages like southeast Asian.

Re: Ziplm: Gzip-Backed Language Model

#18

Post by idbfs »

Cool! I had been thinking about trying this as well, after reading about the idea in one of Cosma Shalizi's notebooks [0]. I'd love to see how something like this performs when "trained" on a corpus the size of the web when given the same kind of computational resources used to train modern LLMs.

[0] http://bactra.org/notebooks/nn-attention-and-transformers.ht...

Re: Ziplm: Gzip-Backed Language Model

#19

Post by canjobear »

usgroup wrote: post #15

Hey author, would you mind explaining the conversion from compression length to probability please? Namely this line: scipy.special.log_softmax(-code_lengths self.conversion (1/temperature)) Your codes are K-ary but this doesn't look like its taken into account ala the README. What is the log(256) conversion factor? What is 1/temperature for?

I’m measuring the length of the gzipped string in bytes, so K=256.

The temperature parameter is there in case anyone wants to play around with it.

Re: Ziplm: Gzip-Backed Language Model

#20

Post by boberoni »

p-e-w wrote: post #12

It should be noted that Moby Dick is (in)famous for its very large and unusual vocabulary. Some lighter reading might yield better results for demonstration purposes. Also, while fairly long for a single novel, the number of words is still minuscule compared to what other LMs are trained on. Using the entire Gutenberg library, or a Wikipedia dump, could improve the quality dramatically. Of course, it doesn't really m…

"An early version of a new project will sometimes be dismissed as a toy. It's a good sign when people do this. That means it has everything a new idea needs except scale, and that tends to follow." [1]

[1] http://www.paulgraham.com/greatwork.html

Post reply on HN