I was curious as to whether this would work. Good to see people trying new things. Because the next is to feed it PGP or AES-256 encrypted data and hand it a symmetric key. Just sort of an attempt to secure what lives in the model itself.
Ziplm: Gzip-Backed Language Model
21–30 of 59 posts
Re: Ziplm: Gzip-Backed Language Model
#22Better write a pull request to have this added to LangChain :^) Eternal, VC money backed, fame awaits!
Re: Ziplm: Gzip-Backed Language Model
#23Hey author, would you mind explaining the conversion from compression length to probability please? Namely this line: scipy.special.log_softmax(-code_lengths self.conversion (1/temperature)) Your codes are K-ary but this doesn't look like its taken into account ala the README. What is the log(256) conversion factor? What is 1/temperature for?
I’m measuring the length of the gzipped string in bytes, so K=256. The temperature parameter is there in case anyone wants to play around with it.
For anyone else wondering how this works out:
1. You want: p(x) ~ K^(-|x|), where K=256.
2. log p(x) ~ log K^(-|x|) = -|x|log K
3. he is using log(softmax) ~ log(e^x)
4. and log(e^(-|x|log(K))) = -|x|*log K as required.
Re: Ziplm: Gzip-Backed Language Model
#24Earlier quoted context omitted.
Maybe this was the inspiration for your project but I just saw a paper that applied compressors as part of a text classification system getting accuracy competitive with BERT. 14 lines of Python. https://aclanthology.org/2023.findings-acl.426/
Yeah, that paper made me wonder how well gzip would work as a language model so I typed up this thing in a coffee shop
Reminds immediately this famous article by Ted Chiang "ChatGPT Is a Blurry JPEG of the Web
" https://www.newyorker.com/tech/annals-of-technology/chatgpt-...
Re: Ziplm: Gzip-Backed Language Model
#25Can someone explain?
Re: Ziplm: Gzip-Backed Language Model
#26[dead]
Re: Ziplm: Gzip-Backed Language Model
#27Oh crud, I made the front page. It should be obvious that this is junk quality as a language model. But it's a cool example of the equivalence between compression codes and probability distributions, so I hope people find it interesting for that reason. You can also get a bit of an intuitive sense for the patterns that gzip and bzip2 pick up on in text (they like repetitive strings).
Repeating this experiment with a more optimal compression algorithm might be interesting too.
Sure! The current best achieved compression for English text (well, specifically, Wikipedia articles) is Fabrice Bellard's nncp, which uses... transformers. http://mattmahoney.net/dc/text.html#1085
Unfortunately I don't think anyone has posted any trained models after running any inputs through it, which could be used to repeat the experiment. I would try it, but I don't have a AVX2 CPU.
However, most research into reducing the model perplexity of text (increasing the compression ratio when the model is used for encoding) uses fixed language models which are learnt offline. Obviously, they use transformers too. You can see some of them here:
https://paperswithcode.com/sota/language-modelling-on-wikite...
However (token) perplexity is only comparable across models if the same tokenizer is used. The above page seems to compare papers using the GPT-2 tokenizer. I would assume that if you controlled for differences in tokenizers by just measuring bits-per-word/byte, GPT-4 without RLHF would achieve the highest compression. It should be obvious that RLHF increases perplexity because it changes the training objective to something other than predicting text.
Re: Ziplm: Gzip-Backed Language Model
#28Perhaps a stupid question, but is this kinda thing related to how Markov chains work?
Re: Ziplm: Gzip-Backed Language Model
#29Previously, a similar work with great performance:
https://news.ycombinator.com/item?id=36707193
Re: Ziplm: Gzip-Backed Language Model
#30Oh crud, I made the front page. It should be obvious that this is junk quality as a language model. But it's a cool example of the equivalence between compression codes and probability distributions, so I hope people find it interesting for that reason. You can also get a bit of an intuitive sense for the patterns that gzip and bzip2 pick up on in text (they like repetitive strings).
Repeating this experiment with a more optimal compression algorithm might be interesting too.
If I understand it correctly, this implementation re-compresses the prefix 256 times, once for each possible byte continuation. It should be possible to save a lot of work by saving the state of the compressor just before the predicted byte, and reusing it.
Also, since gzip works on a limited sliding context window, it's probably pointless to give it a prefix much longer than the window... unless there's something about the algorithm that I'm missing. Same with bzip2 and its block size.