Ziplm: Gzip-Backed Language Model
41–50 of 59 posts
Re: Ziplm: Gzip-Backed Language Model
#42Earlier quoted context omitted.
Yeah, that paper made me wonder how well gzip would work as a language model so I typed up this thing in a coffee shop
Reminds immediately this famous article by Ted Chiang "ChatGPT Is a Blurry JPEG of the Web " https://www.newyorker.com/tech/annals-of-technology/chatgpt-...
Thank you for posting, I enjoyed the article. In true cybernetics fashion, the maximum compression ratio of information seems to be the current grail of our machine learning efforts, whether that is through inferring heuristics or gradually incorporating knowledge about the world.
I'm interested to see where on the information density gradient various organisms exists in their ability to do so..DNA seems awfully wordy, but has proven to be a very dense storage medium.
Re: Ziplm: Gzip-Backed Language Model
#43Conversely, language models can also be used for compression. For example, https://bellard.org/nncp/
Re: Ziplm: Gzip-Backed Language Model
#44What about zstd?
Re: Ziplm: Gzip-Backed Language Model
#45What would happen if each word in "tokenized" to an integer and then you generate tokens instead of characters to produce a string of coherent words instead of random strings? Maybe the answer is obvious but not to me without diving into it at a deeper level. Would be interested to hear anyones thoughts on this.
Re: Ziplm: Gzip-Backed Language Model
#46Earlier quoted context omitted.
"An early version of a new project will sometimes be dismissed as a toy. It's a good sign when people do this. That means it has everything a new idea needs except scale, and that tends to follow." [1] [1] http://www.paulgraham.com/greatwork.html
Strong disagree. In AI, many cool ideas blow up when meeting anything close to real world inputs.
That is the perfect attitude for you then. Please ignore this toy and use only the Webscale tech blessed by google.
Re: Ziplm: Gzip-Backed Language Model
#47Compression based ML approaches and hacks are fascinating.
My colleague recently sent me this: https://arxiv.org/abs/2212.09410
"Less is More: Parameter-Free Text Classification with Gzip
"...We propose a non-parametric alternative to DNNs that's easy, light-weight and universal in text classification: a combination of a simple compressor like gzip with a k-nearest-neighbor classifier. Without any training, pre-training or fine-tuning, our method achieves results that are competitive with non-pretrained deep learning methods on six in-distributed datasets. It even outperforms BERT on all five OOD datasets, including four low-resource languages. Our method also performs particularly well in few-shot settings where labeled data are too scarce for DNNs to achieve a satisfying accuracy."
Re: Ziplm: Gzip-Backed Language Model
#48Oh crud, I made the front page. It should be obvious that this is junk quality as a language model. But it's a cool example of the equivalence between compression codes and probability distributions, so I hope people find it interesting for that reason. You can also get a bit of an intuitive sense for the patterns that gzip and bzip2 pick up on in text (they like repetitive strings).
Maybe this was the inspiration for your project but I just saw a paper that applied compressors as part of a text classification system getting accuracy competitive with BERT. 14 lines of Python. https://aclanthology.org/2023.findings-acl.426/
Yes, but this paper was about classification, not language generation.
Re: Ziplm: Gzip-Backed Language Model
#49Earlier quoted context omitted.
Reminds immediately this famous article by Ted Chiang "ChatGPT Is a Blurry JPEG of the Web " https://www.newyorker.com/tech/annals-of-technology/chatgpt-...
Thank you for posting, I enjoyed the article. In true cybernetics fashion, the maximum compression ratio of information seems to be the current grail of our machine learning efforts, whether that is through inferring heuristics or gradually incorporating knowledge about the world. I'm interested to see where on the information density gradient various organisms exists in their ability to do so..DNA seems awfully word…
There is also an approach to use a LLM and only compress the errors from LLM to the target text. You'd have to add the LLM to the final size, but you could have it already downloaded at the target, in which case I expect nothing will beat it for text compression. Of course it would be very slow.
Re: Ziplm: Gzip-Backed Language Model
#50Earlier quoted context omitted.
I am not sure if it was intentional, but in your "Moby Dick" example, the "alphabet" vocab is missing the letter 'f'.
Ha, yeah. I was just running my finger along the keyboard to type out the string. I just regenerated it, btw, and got a better looking result.
Python can give you all the characters via the constants on the string stdlib: https://docs.python.org/3/library/string.html?highlight=stri..., e.g.:
Python 3.10.6 (main, May 29 2023, 11:10:38) [GCC 11.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import string
>>> string.printable
'0123456789abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ!"#$%&\'()*+,-./:;?@[\\]^_`{|}~ \t\n\r\x0b\x0c'