> Also, these days, one hardly needs a Markov model to generate gibberish; social media provides an ample supply.
I fed 24 years of my blog posts to a Markov model
81–90 of 131 posts
Re: I fed 24 years of my blog posts to a Markov model
#82So, are current LLMs better because artificial neural networks are better predictors than Markov models, or because of the scale of the training data? Just putting it out there..
Markov models usually only predict the next token given the two preceding tokens (trigram model) because the data gets so exceptionally sparse beyond that, that it becomes impossible to make probability estimations (despite back-off, smoothing, etc.). I recommend you to read Bengio et al.’s 2003 paper which describes this issue in more detail and introduces distributional representations (embeddings) in an RNN to avo…
Thanks for the reference and I stand corrected. And yes I had looked at it a long time ago and will give it another read. But I think it is saying that RNNs are a means of approximating a statistical property of a collection of text. That property is what we today think of as "completion"? That is, glorified auto complete, and not "distributed representations" of the world. Would you agree?
Re: I fed 24 years of my blog posts to a Markov model
#83Earlier quoted context omitted.
Markov models usually only predict the next token given the two preceding tokens (trigram model) because the data gets so exceptionally sparse beyond that, that it becomes impossible to make probability estimations (despite back-off, smoothing, etc.). I recommend you to read Bengio et al.’s 2003 paper which describes this issue in more detail and introduces distributional representations (embeddings) in an RNN to avo…
Thanks for the reference and I stand corrected. And yes I had looked at it a long time ago and will give it another read. But I think it is saying that RNNs are a means of approximating a statistical property of a collection of text. That property is what we today think of as "completion"? That is, glorified auto complete, and not "distributed representations" of the world. Would you agree?
distributed representations
Distributional representations, not distributed.
https://en.wikipedia.org/wiki/Distributional_semantics#Distr...
Re: I fed 24 years of my blog posts to a Markov model
#84Here's a quick custom markov page you can have fun with, (all in client) https://aperocky.com/markov/ npm package of the markov model if you just want to play with it on localhost/somewhere else: https://github.com/Aperocky/weighted-markov-generator
Hailo from CPAN (Perl) it's much lighter than any NPM solution.
Re: I fed 24 years of my blog posts to a Markov model
#85I usually have this technical hypothetical discussions with ChatGpt, I can share if you like, me asking him this: aren't LLMs just huge Markov Chains?! And now I see your project... Funny
LLMs are indeed Markov chains. The breakthrough is that we are able to efficiently compute well performing probabilities for many states using ML.
They are definitely not Markov Chains they may, however, be Markov Models. There's a difference between MC and MM.
Re: I fed 24 years of my blog posts to a Markov model
#86Earlier quoted context omitted.
LLMs are not Markov Chains unless you contort the meaning of a Markov Model State so much you could even include the human brain.
Well LLMs aren't human brains, unless you contort the definition of matrix algebra so much you could even include them.
QM and GR can be written as matrix algebra, atoms and electrons are QM, chemistry is atoms and electrons, biology is chemistry, brains are biology.
An LLM could be implemented with a Markov chain, but the naïve matrix is ((vocab size)^(context length))^2, which is far too big to fit in this universe.
Like, the Bekenstein bound means writing the transition matrix for an LLM with just 4k context (and 50k vocabulary) at just one bit resolution, the first row (out of a bit more than 10^18795 rows) ends up with a black hole >10^9800 times larger than the observable universe.
Re: I fed 24 years of my blog posts to a Markov model
#87Quick test for Perl users (so anyone there with a Unix-like).
Run these as a NON root user:
cpanm -n local::lib
cpanm -n Hailo
~/perl5/bin/hailo -E Scored -t corpus.txt -b brain.brn
~/perl5/bin/hailo -b brain.brn
I forgot to put the '-E' flag in my previous comments, so here it is. It's to select a more 'complex' engine, so the text output looks less gibberish.
Re: I fed 24 years of my blog posts to a Markov model
#88https://cdn.cs50.net/ai/2023/x/lectures/6/src6/markov/#
This is a nice Markov text generator.
Re: I fed 24 years of my blog posts to a Markov model
#89I recall a Markov chain bot on IRC in the mid 2000s. I didn't see anything better until gpt came along!
Perhaps you are thinking of megahal https://homepage.kranzky.com/megahal/Index.html or if a bit later in the millennium, cobe https://teichman.org/blog/
Re: I fed 24 years of my blog posts to a Markov model
#90Earlier quoted context omitted.
LLMs are indeed Markov chains. The breakthrough is that we are able to efficiently compute well performing probabilities for many states using ML.
They are definitely not Markov Chains they may, however, be Markov Models . There's a difference between MC and MM.
What do you mean? The states are fully observable (current array of tokens), and using an LLM we calculate the probabilities of moving between them. What is not MC about this?