91–100 of 177 posts
Earlier quoted context omitted.
Yeah, and Markov chains aren't much smarter than... matrix multiplication. A lot of stuff have been built with it, but these pesky c++ libs like lapack — such a bother to use! Imagine a world where a good infrastructure have been built around it — matrix multiplication certainly would be a blast!
What are ANN but fancy matrix multiplication?
It could be me not understanding something but: > Since there is a 25% probability Alice is at the grocery store, we multiply that with the probility of her transitioning to the Planetarium: 25% ∗ 75% Shouldn't this be 25% * 70%?
-----
I'd rather do 0.25 * 0.75
Earlier quoted context omitted.
A LLM is a Markov chain with billions of associations and weights. A Makov chain is an LLM of maybe a few dozen associations and weights (so an LM, without the first L). The difference is in the data structure and the size of the atoms/n-grams. The data structure Markov chain implementations use is not efficient for billions of parameters, either in storage or in processing. But the idea is the same: give a likely ne…
“A human brain is just like a dog’s brain, only with more neural pathways.” True, perhaps, but largely pointless: at some point neural complexity results in a difference of kind, not of degree. I’d argue the same is true of LLMs vs simpler models like Markov chains.
Earlier quoted context omitted.
>However, explaining that a LLMs are really just iterated next-word prediction based on a statistical model of the preceding words is something that most people can grok, and in a useful way: in my experience, it actually helps give people a useful intuition for why and how models hallucinate and what kind of things they're good/bad at. At risk of showing my deficient understanding: that isn't actually true, is it?[1…
Predicting subsequent text is pretty much exactly what they do. Lots of very cool engineering that’s a real feat, but at its core it’s argmax(P(token|token,corpus)): https://github.com/facebookresearch/llama/blob/main/llama/ge... The engineering feats are up there with anything, but it’s a next token predictor.
This is what I've been saying to people, LLMs aren't much smarter than a Markov Chain - a lot of twitter bots and earlier chat agents were driven by them. What I do see though is the infrastructure around LLMs making a difference.
> This is what I've been saying to people, LLMs aren't much smarter than a Markov Chain Then you're doing some combination of grossly overestimating the latter and/or underestimating the former. A Markov Chain is a (small) language model with a context length of 1, never more, that's a definitional requirement of a Markov process.
As an example, de Bruijn graphs have been widely used in bioinformatics. There are many definitions, but the key idea is that nodes are strings of length k, and there may be an edge from u to v if the length k-1 suffix of u is the same as the length k-1 prefix of v. It's trivial to turn a de Bruijn graph into a Markov chain with a context of k symbols.
If you have the Burrows-Wheeler transform of the training data, you can use it to simulate a Markov chain over a de Bruijn graph with any context length. You can also change the context length on the fly. I toyed a little bit with that idea when I was a grad student ~15 years ago, but nothing really came out of it. Building the model was already cheap back then, while getting data was the real bottleneck.
LLMs are basically lossy approximations of Markov chains. If you have a small accurate approximation of a larger model, the approximate model probably has the ability to generalize. And it turns out that good generalization over textual data looks like intelligent behavior.
Earlier quoted context omitted.
A typical demonstration markov chain probably has a length of around 3. A typical recent LLM probably has more than three billion parameters. That's not precisely apppes to apples, but the LLM is certainly vastly more complicated.
The way you describe it, it doesn't seem much more complicated to me, from a “how does it work” perspective, just way bigger.
With a Markov chain, the statistics are as simple as a mapping of n-grams to the number of times it appears in the corpus.
With a LLM, the statistics are the result of 50 years of research in neural network architectures, terabytes of training data, and many millions of dollars worth of hardware, along with the teams to build and manage all the data pipelines.
So yes, much more complicated.
Earlier quoted context omitted.
Predicting subsequent text is pretty much exactly what they do. Lots of very cool engineering that’s a real feat, but at its core it’s argmax(P(token|token,corpus)): https://github.com/facebookresearch/llama/blob/main/llama/ge... The engineering feats are up there with anything, but it’s a next token predictor.
Did read the part about king - man + woman = queen? How is that a next-token predictor?
Now, how exactly the LLM is computing this prediction remains poorly understood.
Sure, LLMs are like 1024-gram Markov chains (or whatever the context limit is). But there are problems: 1) the transition matrix is far too huge to represent, and 2) it treats all pairs of 1024-grams as completely different, even if the first 1023 words are the same. Function approximation solves both issues, and the Transformer is the best function clas we've found so far.
LLMs are a reincarnation of Markov Chains, which, by the way, were surprisingly underutilized in the past.