Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Markov Chains are the Original Language Models

elijahpotter.dev

Post reply on HN
Print view View original

81–90 of 177 posts

Re: Markov Chains are the Original Language Models

#81

Post by pyinstallwoes »

rwmj wrote: post #75
thfuran wrote: post #70

Earlier quoted context omitted.

A typical demonstration markov chain probably has a length of around 3. A typical recent LLM probably has more than three billion parameters. That's not precisely apppes to apples, but the LLM is certainly vastly more complicated.

Number of parameters is not the difference. A Markov chain can easily be a multi-dimensional matrix with millions of entries. The significant difference is that a length 3 Markov chain can only ever find connections between 3 adjacent symbols (words, usually). LLMs seem to be able to find and connect abstract concepts at a very long and variable distances in the input. Nevertheless I agree with the premise of the pos…

Seemingly? Is there not a direct technical reason to compare?

Re: Markov Chains are the Original Language Models

#82

Post by pyinstallwoes »

SilasX wrote: post #69
lukev wrote: post #52

What's actually happening in a LLM is many orders of magnitude more complex than a Markov chain. However, I agree that they're an amazing pedagogical tool for the basic principles of how a LLM works, even to a non-technical audience. Many people try to "explain" LLMs starting with the principles of neural networks. This rarely works well: there are some significant conceptual leaps required. However, explaining that…

>However, explaining that a LLMs are really just iterated next-word prediction based on a statistical model of the preceding words is something that most people can grok, and in a useful way: in my experience, it actually helps give people a useful intuition for why and how models hallucinate and what kind of things they're good/bad at. At risk of showing my deficient understanding: that isn't actually true, is it?[1…

What you described was the vector space utility and mathematics which is the result of encoding it into a dimensional space (latent?).

SOTA LLM use vector space dimensional embeddings in order to utilize a coordinate based intelligence “for free” with a vector space prediction mechanism on the index of context.

Re: Markov Chains are the Original Language Models

#83

Post by pyinstallwoes »

max_ wrote: post #80
intalentive wrote: post #78

Claude Shannon showed you could use n-grams to represent language. Then Chomsky invented automata / formal languages theory to show why it couldn’t work. IIRC transformers are at the very bottom of the Chomsky hierarchy, and yet they are clearly able to master English grammar. What gives?

Have they really mastered language or are they mimicking it. Do Submarines swim like fish?

If it quacks like a duck, walks like a duck, is it a duck?

Re: Markov Chains are the Original Language Models

#84

Post by swyx »

chilipepperhott wrote: post #11
jairuhme wrote: post #7

Earlier quoted context omitted.

I'm also getting a client side error where the demo should be? Not sure if its just on my end.

My goodness. Thank you for letting me know. I'll look into it right away.

> Uncaught TypeError: Cannot read properties of null (reading 'getContext') at mq_js_bundle.js:1:90

classic

Re: Markov Chains are the Original Language Models

#86

Post by lsy »

Though it seems like there is some sensitivity around the comparison of LLMs to Markov chains, and certainly an LLM is not generated in the same way, it is pretty accurate that an LLM could be represented by a sufficiently (ie very) complex Markov chain. The states of the chain would not be the tokens themselves, as in this example, but the total context window vector of N input tokens, which would fan out to states consisting of the new context N[1:] + A, where A is any of the tokens sampled from the resulting probability distribution, and the transition probabilities are just drawn from that same distribution according to the temperature settings.

You could even do some very hand-wavy math on how staggeringly complex the resulting Markov chain would get: BERT for example has a token vocabulary of 30,000 and a context window of 512 tokens. So the number of possible states would be 30,000^512, or ~1.9 x 10^2292, with each of those having a max fan out to 30,000 other states. So clearly the LLM is a more compact representation of the concept.

Re: Markov Chains are the Original Language Models

#87

Post by pyinstallwoes »

CamperBob2 wrote: post #42
thwarted wrote: post #23

Earlier quoted context omitted.

A LLM is a Markov chain with billions of associations and weights. A Makov chain is an LLM of maybe a few dozen associations and weights (so an LM, without the first L). The difference is in the data structure and the size of the atoms/n-grams. The data structure Markov chain implementations use is not efficient for billions of parameters, either in storage or in processing. But the idea is the same: give a likely ne…

It's not just window size. It's the difference between syntax and semantics. A Markov model, by definition, works only with literal token histories. It can't participate meaningfully in a conversation unless the user happens to employ token sequences that the model has seen before (ideally multiple times.) An LLM can explain why it's not just a Markov model, but the converse isn't true. Now, if you were to add high-d…

I like: Intelligence is compressing information into irreducible representation.

Which leads to a wonderful tongue-in-cheek contraindication: representation types such as a particular model, when it’s complexity increases, especially via edge cases, it is then a result of agentic anti-intelligence.

That is to say, anything that increases in complexity without being refactored is a sign of a lack of intelligence or worse.

And any sense of information that is impossible to be reduced further while maintaining equal or more expressibility are signs of maximum agentic intelligence.

Re: Markov Chains are the Original Language Models

#88

Post by SilasX »

pyinstallwoes wrote: post #82
SilasX wrote: post #69

Earlier quoted context omitted.

>However, explaining that a LLMs are really just iterated next-word prediction based on a statistical model of the preceding words is something that most people can grok, and in a useful way: in my experience, it actually helps give people a useful intuition for why and how models hallucinate and what kind of things they're good/bad at. At risk of showing my deficient understanding: that isn't actually true, is it?[1…

What you described was the vector space utility and mathematics which is the result of encoding it into a dimensional space (latent?). SOTA LLM use vector space dimensional embeddings in order to utilize a coordinate based intelligence “for free” with a vector space prediction mechanism on the index of context.

Thanks, but can you clarify which confusion in my comment you're untangling there?

Re: Markov Chains are the Original Language Models

#89

Post by benreesman »

SilasX wrote: post #69
lukev wrote: post #52

What's actually happening in a LLM is many orders of magnitude more complex than a Markov chain. However, I agree that they're an amazing pedagogical tool for the basic principles of how a LLM works, even to a non-technical audience. Many people try to "explain" LLMs starting with the principles of neural networks. This rarely works well: there are some significant conceptual leaps required. However, explaining that…

>However, explaining that a LLMs are really just iterated next-word prediction based on a statistical model of the preceding words is something that most people can grok, and in a useful way: in my experience, it actually helps give people a useful intuition for why and how models hallucinate and what kind of things they're good/bad at. At risk of showing my deficient understanding: that isn't actually true, is it?[1…

Predicting subsequent text is pretty much exactly what they do. Lots of very cool engineering that’s a real feat, but at its core it’s argmax(P(token|token,corpus)):

https://github.com/facebookresearch/llama/blob/main/llama/ge...

The engineering feats are up there with anything, but it’s a next token predictor.

Re: Markov Chains are the Original Language Models

#90

Post by lukev »

wongarsu wrote: post #67
lukev wrote: post #52

What's actually happening in a LLM is many orders of magnitude more complex than a Markov chain. However, I agree that they're an amazing pedagogical tool for the basic principles of how a LLM works, even to a non-technical audience. Many people try to "explain" LLMs starting with the principles of neural networks. This rarely works well: there are some significant conceptual leaps required. However, explaining that…

They are an especially useful tool right now, that might become less valuable as we get better at building LLMs. In principle the inner working of an LLM can be anything from a Markov-chain-like predictor to a beyond-human intelligence. Token prediction is the input/output format we chose, but you could communicate with a human in the same format and the human would show human-level intelligence. What makes Markov ch…

> In principle the inner working of an LLM can be anything from a Markov-chain-like predictor to a beyond-human intelligence.

I'm afraid I have to disagree. Next-token prediction isn't just the interface we use for LLMs, it is fundamentally what they are, to the very core. The training and loss function of the foundation models are completely oriented towards next-token accuracy.

Reasonable people can disagree about emergent behavior and if/how much the model is "planning ahead" in its weights (and what that could even mean) but it is emphatically not the case that the "next token" model is "just an interface". The analogy to human thought isn't accurate at all: we have our own recursive/iterative thought process, short and long term memory, decision making loops, etc.

A LLMs has no "thought" outside of next-token prediction and no working memory aside from its context window. We don't fully understand all the emergent behavior of a transformer model but we definitely understand exactly what's happening at the mechanical level: each token is determined, one at a time, by solving an extremely complex but deterministic equation in which the model weights are coefficients, and the output of which is a probability distribution over the next token.

There's no hidden intelligence or man behind the curtain. Whatever a LLM can do, next token prediction is how it does it.

Post reply on HN