Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Markov Chains are the Original Language Models

elijahpotter.dev

Post reply on HN
Print view View original

121–130 of 177 posts

Re: Markov Chains are the Original Language Models

#121

Post by csomar »

SilasX wrote: post #69
lukev wrote: post #52

What's actually happening in a LLM is many orders of magnitude more complex than a Markov chain. However, I agree that they're an amazing pedagogical tool for the basic principles of how a LLM works, even to a non-technical audience. Many people try to "explain" LLMs starting with the principles of neural networks. This rarely works well: there are some significant conceptual leaps required. However, explaining that…

>However, explaining that a LLMs are really just iterated next-word prediction based on a statistical model of the preceding words is something that most people can grok, and in a useful way: in my experience, it actually helps give people a useful intuition for why and how models hallucinate and what kind of things they're good/bad at. At risk of showing my deficient understanding: that isn't actually true, is it?[1…

> That primitive is not present in the Markov predictor example

It is. You need to go multi-dimensional; that's where "intelligence" emerges.

Re: Markov Chains are the Original Language Models

#122

Post by kromem »

benreesman wrote: post #89
SilasX wrote: post #69

Earlier quoted context omitted.

>However, explaining that a LLMs are really just iterated next-word prediction based on a statistical model of the preceding words is something that most people can grok, and in a useful way: in my experience, it actually helps give people a useful intuition for why and how models hallucinate and what kind of things they're good/bad at. At risk of showing my deficient understanding: that isn't actually true, is it?[1…

Predicting subsequent text is pretty much exactly what they do. Lots of very cool engineering that’s a real feat, but at its core it’s argmax(P(token|token,corpus)): https://github.com/facebookresearch/llama/blob/main/llama/ge... The engineering feats are up there with anything, but it’s a next token predictor.

Predicting the next token is how they are trained. But how they operate to achieve that goal is much, much more complicated than the principle of a Markov chain.

In theory, you are the byproduct of 'training' to survive and reproduce. Humans are very good at that task.

But their net capabilities developed to succeed at it extend far beyond the scope of the task alone.

Re: Markov Chains are the Original Language Models

#123

Post by kromem »

lsy wrote: post #86

Though it seems like there is some sensitivity around the comparison of LLMs to Markov chains, and certainly an LLM is not generated in the same way, it is pretty accurate that an LLM could be represented by a sufficiently (ie very) complex Markov chain. The states of the chain would not be the tokens themselves, as in this example, but the total context window vector of N input tokens, which would fan out to states…

It would be more accurate to say that a Markov chain is an example of a method that would perform relatively well at the same training task as a LLM.

So too, might a human trying to predict the next tokens.

But a human and a Markov chain are not the same underlying process to achieve next token prediction, and neither is the same underlying process as a LLM.

Re: Markov Chains are the Original Language Models

#124

Post by Jensson »

kromem wrote: post #123
lsy wrote: post #86

Though it seems like there is some sensitivity around the comparison of LLMs to Markov chains, and certainly an LLM is not generated in the same way, it is pretty accurate that an LLM could be represented by a sufficiently (ie very) complex Markov chain. The states of the chain would not be the tokens themselves, as in this example, but the total context window vector of N input tokens, which would fan out to states…

It would be more accurate to say that a Markov chain is an example of a method that would perform relatively well at the same training task as a LLM. So too, might a human trying to predict the next tokens. But a human and a Markov chain are not the same underlying process to achieve next token prediction, and neither is the same underlying process as a LLM.

LLM are markov chains, a markov chain is a general concept and not just a text model technique. You must be thinking about the very simple markov chain models we had before where you just predicted the next word by looking up sentences with the same preceding words and picking a random of those words, that is also a markov chain just like LLM but a much simpler one, you are right LLMs aren't like that but they are still markov chains with the same kind of inputs and outputs as the old ones.

Re: Markov Chains are the Original Language Models

#125

Post by Jensson »

max_ wrote: post #77
feoren wrote: post #73

Earlier quoted context omitted.

> that they are one of the simplest and most intuitive ways to write code that learns from input data Linear regression is significantly simpler and also learns from input data.

Markov chains are basically lookup tables. I don't see how linear regression is simpler that that.

Markov chains aren't lookup tables... Markov chains is the description of a process where you generate the next step by looking at preceding steps, that can be a lookup table but it can also be more complex like modern LLMs.

Re: Markov Chains are the Original Language Models

#126

Post by Jensson »

candiodari wrote: post #59
causalmodels wrote: post #40

Earlier quoted context omitted.

An LLM is a Markov chain in the same sense that a cat is a tiger, technically true but it misses the qualia.

It's not. There's fundamental architectural differences that couldn't be bigger. A better comparison would be that it's like a windup toy versus a group of humans moving an entire civilization. They both move along a distance, but just listing the systems that the human group has that the windup toy doesn't is too long to fit on a page.

> It's not. There's fundamental architectural differences that couldn't be bigger.

LLM architecture is a markov chain to the core. It isn't a lookup table like old markov chains but it is still a markov chain: next word prediction based on previous words.

Re: Markov Chains are the Original Language Models

#127

Post by Jensson »

esafak wrote: post #107
PheonixPharts wrote: post #15

Earlier quoted context omitted.

LLMs are Markov chains in latent space , it's the latent representation that give them their power, but ultimately there's not as much difference as one would suspect.

They're different because Markov models are stateless whereas LLMs are stateful. https://en.wikipedia.org/wiki/Markov_property

Current LLMs are stateless as far as we know, their state when computing a new token is only the preceding text tokens, they don't store any metadata or save state from the previous calculations.

Re: Markov Chains are the Original Language Models

#128

Post by Jensson »

pyinstallwoes wrote: post #83
max_ wrote: post #80

Earlier quoted context omitted.

Have they really mastered language or are they mimicking it. Do Submarines swim like fish?

If it quacks like a duck, walks like a duck, is it a duck?

We haven't made machines that replace animals. A submarine can't do quick precision movement to pick up tiny objects like a fish, a plane can't do quick precision movement to fly through a forest and land gracefully on a branch like a bird.

The current machine intelligence is still clumsy like that, it can solve some problems but it is still far from the real thing.

Re: Markov Chains are the Original Language Models

#129

Post by tarsinge »

lukev wrote: post #96
skrebbel wrote: post #74

Earlier quoted context omitted.

The way you describe it, it doesn't seem much more complicated to me, from a “how does it work” perspective, just way bigger.

The overall structure is the same as in "use statistics to predict the next token." With a Markov chain, the statistics are as simple as a mapping of n-grams to the number of times it appears in the corpus. With a LLM, the statistics are the result of 50 years of research in neural network architectures, terabytes of training data, and many millions of dollars worth of hardware, along with the teams to build and mana…

You can have very complex calculations and a simple output. The complexity of the process to find the weights is not necessarily the same complexity than the process using them. The numbers don't get suddenly special because of how they were calculated (like 42 is just the number 42 even after 7 million years of calculations).

Re: Markov Chains are the Original Language Models

#130

Post by withinboredom »

SilasX wrote: post #111
svachalek wrote: post #102

Earlier quoted context omitted.

With the caveat that I'm not an LLM expert here but have read up on some of this... What's basically going on is that the LLM has "read" vast amounts of text and classified words in all kinds of dimensions. As it goes from word to word it's predicting based on all of those dimensions, rather than the Markov chain's simple probabilities. So it knows "king" has high values of masculinity and authority, for example, and…

So you're saying that the word2vec king/queen result was the result of feeding the verbatim text "king+man-woman", and that text being continued as "queen"? I assumed that was more the result of doing math on the properties the model generated for the tokens king, man, queen, etc. And in that case, why does "Write me a limerick " result in ChatGPT producing limericks, when vanishingly few of the limericks in the sour…

You’d also get text that has very little to do with the training material from any statistical model. A prediction may have nothing to do with the past and it might be wrong or right. For example, the weather forecast said it will rain all day, but the sun is up and bright in the sky without a cloud in sight.

The model knows what a limerick is, from the source material. It knows what your criteria is from the source material. It can predict what someone would say given that prompt.

Humans also do this. I’m usually one or two words ahead of the person I’m speaking to, sometimes even entire paragraphs ahead if I’m paying full attention. My dreams give me unrealistic situations to explore new ways of dealing with them. When I write code, I have a pretty good idea of what I’m going to write before I write it.

The main difference between a human and an LLM, is that a human has no limit. A human will still continue when overwhelmed with data, usually by shedding unimportant data. An LLM will just tell you it’s too much data. Smart humans won’t just shed the data, but “mark” it mentally as potentially important in the future and come back to it once a deeper understanding is achieved.

There are other, smaller differences as well, but that is the biggest, most annoying one, so far.

Post reply on HN