Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Ziplm: Gzip-Backed Language Model

github.com

Post reply on HN
Print view View original

31–40 of 59 posts

Re: Ziplm: Gzip-Backed Language Model

#31

Post by vintermann »

canjobear wrote: post #5
ronsor wrote: post #3

Earlier quoted context omitted.

I think you would get slightly better generations from the model if the input data were tokenized and token IDs were sampled instead.

Possibly. It would definitely make it even slower though. To sample from the model, if you have a prompt/training data c, you have to compute the compressed length of all strings cx for tokens x in your vocabulary. If you have a long prompt c (like 1 million lines of wikipedia) and a big vocabulary of tokens (like 100,000 different words), you're going to have a bad time.

It should be possible to save and reuse the state of the compressor right before the final byte.

Re: Ziplm: Gzip-Backed Language Model

#32

Post by manmal »

canjobear wrote: post #2

Oh crud, I made the front page. It should be obvious that this is junk quality as a language model. But it's a cool example of the equivalence between compression codes and probability distributions, so I hope people find it interesting for that reason. You can also get a bit of an intuitive sense for the patterns that gzip and bzip2 pick up on in text (they like repetitive strings).

A maybe dumb question for understanding: does a longer output imply lower probability of the input?

Re: Ziplm: Gzip-Backed Language Model

#34

Post by OriPekelman »

djxfade wrote: post #28

Perhaps a stupid question, but is this kinda thing related to how Markov chains work?

Basically "Information theoretically" compression is a measure of informational distance. So basically if text A and Text B contactenated together, compress better than text A and text C it means A and B repeat more patterns. and are closer. All we need are some distance functions .. in a way you can think about it like Levenhstein distance but that can take into account inputs with very different sizes, repetitions, changes in order big inserts etc...

Reminds me of a mostly joke ruby project I did a decade ago https://github.com/oripekelman/simple_similarity

Re: Ziplm: Gzip-Backed Language Model

#35

Post by DonHopkins »

sam_lowry_ wrote: post #24
canjobear wrote: post #11

Earlier quoted context omitted.

Yeah, that paper made me wonder how well gzip would work as a language model so I typed up this thing in a coffee shop

Reminds immediately this famous article by Ted Chiang "ChatGPT Is a Blurry JPEG of the Web " https://www.newyorker.com/tech/annals-of-technology/chatgpt-...

I'm looking forward to Progressive ChatGPT, that immediately answers your question with an thoughtfully chosen emoji, that expands into an irresistible clickbait title, that expands into a viral tweet, that expands into a fascinating forum comment, that expands into a ranting and raving blog post, that expands into a ponderous wikipedia page, that expands into a voluminous e-book, that expands into ...

Re: Ziplm: Gzip-Backed Language Model

#36

Post by Vetch »

idbfs has posted this link already but did not explain that Shalizi provides a deep theoretical explanation for why universal source coding (does not require information about symbol distribution or statistics) such as Lempel ziv derived compression algorithms can serve as powerful language models if practical restrictions on them such as dictionary and input corpus size are lifted.

This is a good example of how old methods can be pushed quite far if similar resources were devoted to them. Who knows, they might even posses advantages hitherto unmet due to a lack of exploring at larger scales.

That said, Transformers have a number of practical advantages. The learned projection matrices in attention lend Transformers a dynamic adaptability with respect to learned patterns that help make them programmable by their context, able to work out patterns present in context zero shot and on the fly. gzip based language models will be limited to their dictionary of patterns. The underlying vector space of neural language models also makes semantics more readily learnable (driving novel synthesis such as neologisms and more) while feed forward layers can learn a large range of computations.

http://bactra.org/notebooks/nn-attention-and-transformers.ht...

Re: Ziplm: Gzip-Backed Language Model

#38

Post by optimalsolver »

boberoni wrote: post #20
p-e-w wrote: post #12

It should be noted that Moby Dick is (in)famous for its very large and unusual vocabulary. Some lighter reading might yield better results for demonstration purposes. Also, while fairly long for a single novel, the number of words is still minuscule compared to what other LMs are trained on. Using the entire Gutenberg library, or a Wikipedia dump, could improve the quality dramatically. Of course, it doesn't really m…

"An early version of a new project will sometimes be dismissed as a toy. It's a good sign when people do this. That means it has everything a new idea needs except scale, and that tends to follow." [1] [1] http://www.paulgraham.com/greatwork.html

Strong disagree. In AI, many cool ideas blow up when meeting anything close to real world inputs.

Re: Ziplm: Gzip-Backed Language Model

#40

Post by usgroup »

manmal wrote: post #32
canjobear wrote: post #2

Oh crud, I made the front page. It should be obvious that this is junk quality as a language model. But it's a cool example of the equivalence between compression codes and probability distributions, so I hope people find it interesting for that reason. You can also get a bit of an intuitive sense for the patterns that gzip and bzip2 pick up on in text (they like repetitive strings).

A maybe dumb question for understanding: does a longer output imply lower probability of the input?

its not a dumb question -- optimal coding in theory assigns codes of shorter length to more frequently occurring tokens hence by design longer codes imply lower probabilities: that's how compression is achieved.
Post reply on HN