If you like Markov Chains, Mamba is the architecture you should be looking into today.
Markov Chains are the Original Language Models
31–40 of 177 posts
Re: Markov Chains are the Original Language Models
#32This is what I've been saying to people, LLMs aren't much smarter than a Markov Chain - a lot of twitter bots and earlier chat agents were driven by them. What I do see though is the infrastructure around LLMs making a difference.
Yeah, and Markov chains aren't much smarter than... matrix multiplication. A lot of stuff have been built with it, but these pesky c++ libs like lapack — such a bother to use! Imagine a world where a good infrastructure have been built around it — matrix multiplication certainly would be a blast!
Re: Markov Chains are the Original Language Models
#33This is what I've been saying to people, LLMs aren't much smarter than a Markov Chain - a lot of twitter bots and earlier chat agents were driven by them. What I do see though is the infrastructure around LLMs making a difference.
I've never seen a Markov chain do anything like GPT4. I'm not sure how you can say with a straight face they are basically the same.
[dead]
Re: Markov Chains are the Original Language Models
#34> This a scratch text area to test the autocomplete in another moment down would be four thousand miles i wish you know please ma am is this new zealand or conversations in another moment down stairs how funny it ^ to me it’s really a stretch that someone could read that and think “markov chains are practically an llm!” They’re pretty limited, and everyone working on building LLMs knows about them and has probably us…
> This is where I am right now. Want to go back to my roots. Some people work on old cars, even though they are less efficient. At the same time though, they are more fun to work on than new cars. I've decided to look into Markov chains.
Not sure there’s another side to your argument - we all agree :).
To the author: keep the faith and the excitement if you can, the actual applications of this technology (beyond chatbots) are just getting started.
Re: Markov Chains are the Original Language Models
#35Earlier quoted context omitted.
I've never seen a Markov chain do anything like GPT4. I'm not sure how you can say with a straight face they are basically the same.
No one has spent 100M on training Markov chains.
For the first few years of smartphones markov chains were used for autocorrect.
Re: Markov Chains are the Original Language Models
#36This is what I've been saying to people, LLMs aren't much smarter than a Markov Chain - a lot of twitter bots and earlier chat agents were driven by them. What I do see though is the infrastructure around LLMs making a difference.
Yeah, and Markov chains aren't much smarter than... matrix multiplication. A lot of stuff have been built with it, but these pesky c++ libs like lapack — such a bother to use! Imagine a world where a good infrastructure have been built around it — matrix multiplication certainly would be a blast!
What are ANN but fancy matrix multiplication?
Re: Markov Chains are the Original Language Models
#37Earlier quoted context omitted.
I've never seen a Markov chain do anything like GPT4. I'm not sure how you can say with a straight face they are basically the same.
No one has spent 100M on training Markov chains.
Not 100M I guess, but this is fresh from today's arXiv:
"Infini-gram: Scaling Unbounded n-gram Language Models to a Trillion Tokens"
Re: Markov Chains are the Original Language Models
#38It's true that Markov chains are very limited in their capabilities. But one thing I love about them is that they are one of the simplest and most intuitive ways to write code that *learns* from input data.
If you're never written something that *learns*, try it out! Here's a very primitive one I wrote recently to explain the basic idea and explains it along the way.
https://github.com/unoti/markov-basics/blob/main/markov-basi...
This starts with generating US city names, and ends with generating text based on the Dungeons and Dragons DM guide.
Re: Markov Chains are the Original Language Models
#39Earlier quoted context omitted.
The whole point of Transformers is that they broke the Markov assumption (i.e., that the next token probability is strictly conditioned on a window of N preceding tokens).
That's not actually true, they still have a fixed history window. The idea that transformers capture through the attention mechanism is that not all past tokens are created equal, and that the importance of tokens in that history window depends on what they are.
They also scale differently - Markov Chains scale exponentially with the size of the window, while transformers scale quadratically. So in fact transformers are really more exponentially more efficient, though without bound on resources their capabilities are a strict subset of that of Markov chain.
Re: Markov Chains are the Original Language Models
#40Earlier quoted context omitted.
I've never seen a Markov chain do anything like GPT4. I'm not sure how you can say with a straight face they are basically the same.
A LLM is a Markov chain with billions of associations and weights. A Makov chain is an LLM of maybe a few dozen associations and weights (so an LM, without the first L). The difference is in the data structure and the size of the atoms/n-grams. The data structure Markov chain implementations use is not efficient for billions of parameters, either in storage or in processing. But the idea is the same: give a likely ne…
An LLM is a Markov chain in the same sense that a cat is a tiger, technically true but it misses the qualia.