111–120 of 131 posts
Cool article, it got me to play around with Markov models, too! I first did a Markov model over plain characters. > Itheve whe oiv v f vidleared ods alat akn atr. s m w bl po ar 20 Using pairs of consecutive characters (order-2 Markov model) helps, but not much: > I hateregratics.pyth fwd-i-sed wor is wors.py Triplets (order 3) are a bit better: > I Fed tooks of the say, I just train. All can beconsist answer efferes…
Earlier quoted context omitted.
Learnable? What does that mean?
There are various notions of this. The most basic/naive one is where one can estimate the unknown parameters of the model given example token streams generated by the model.
I think this is more correctly described as a trigram model than a Markov model, if it would naturally expand to 4-grams when they were available, etc, the text would look more coherent Iirc there was some research on "infini-gram", that is a very large ngram model, that allegedly got performance close to LLMs in some domains a couple years back
It achieved state-of-the-art performance at tasks like spelling correction at the time. However, unlike an LLM, it can't generalize at all; if an n-gram isn't in the training corpus it has no idea how to handle it.
https://research.google/blog/all-our-n-gram-are-belong-to-yo...
Earlier quoted context omitted.
There are various notions of this. The most basic/naive one is where one can estimate the unknown parameters of the model given example token streams generated by the model.
So learnable, in this context, rhymes with reverse-engineerable?
Identifiability means that out of all possible models, you can learn the correct one given enough samples.causal identifiability has some other connotations
See here https://causalai.net/r80.pdf as a good start (a nose in a causal graph is Markov given its parents, and a k-step Markov chain is a k-layer causal dag)
Earlier quoted context omitted.
Well LLMs aren't human brains, unless you contort the definition of matrix algebra so much you could even include them.
QM and GR can be written as matrix algebra, atoms and electrons are QM, chemistry is atoms and electrons, biology is chemistry, brains are biology. An LLM could be implemented with a Markov chain, but the naïve matrix is ((vocab size)^(context length))^2, which is far too big to fit in this universe. Like, the Bekenstein bound means writing the transition matrix for an LLM with just 4k context (and 50k vocabulary) at…
The case for brain states and ideas is similar to QM and massive objects. While certain metaphysical presuppositions might hold that everything must be physical and describable by models for physical things, science, which should eschew metaphysical assumptions, has not shown that to be the case.
Cool article, it got me to play around with Markov models, too! I first did a Markov model over plain characters. > Itheve whe oiv v f vidleared ods alat akn atr. s m w bl po ar 20 Using pairs of consecutive characters (order-2 Markov model) helps, but not much: > I hateregratics.pyth fwd-i-sed wor is wors.py Triplets (order 3) are a bit better: > I Fed tooks of the say, I just train. All can beconsist answer efferes…
It would make sense to repeat the experiment with the tokenizer of an LLM.
I did something similar many years ago. I fed about half a million words (two decades of mostly fantasy and science fiction writing) into a Markov model that could generate text using a “gram slider” ranging from 2-grams to 5-grams. I used it as a kind of “dream well” whenever I wanted to draw some muse from the same deep spring. It felt like a spiritual successor to what I used to do as a kid: flipping to a random p…
[0] https://archive.org/details/Babble_1020, https://vetusware.com/download/Babble%21%202.0/?id=11924