Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Markov Chains are the Original Language Models

elijahpotter.dev

Post reply on HN
Print view View original

51–60 of 177 posts

Re: Markov Chains are the Original Language Models

#51

Post by arketyp »

nonameiguess wrote: post #47

I have no idea if the author is aware, but it's worth noting why a Markov chain has the name and what the difference is from other probabilistic models. The Markov property states that the probability distribution of the next state in some system depends only upon the current state. Obviously, language does not have this property and this has been known from the start, but Markov models are extremely computationally…

I've seen Markov models mentioned a lot in these context, and my generous take has always been that something like stacked Markov models is meant. At each abstraction layer, the state is conditioned only by the previous abstract concept. At the lowest level the states would be the sequence of tokens; higher up it's concepts like turn of events in a plot. I don't think this often proposed idea of hierarchy is sufficient to describe LLMs or human cognition, but it strikes at some essence about parsimony, efficient representation, and local computation.

Re: Markov Chains are the Original Language Models

#52

Post by lukev »

What's actually happening in a LLM is many orders of magnitude more complex than a Markov chain. However, I agree that they're an amazing pedagogical tool for the basic principles of how a LLM works, even to a non-technical audience.

Many people try to "explain" LLMs starting with the principles of neural networks. This rarely works well: there are some significant conceptual leaps required.

However, explaining that a LLMs are really just iterated next-word prediction based on a statistical model of the preceding words is something that most people can grok, and in a useful way: in my experience, it actually helps give people a useful intuition for why and how models hallucinate and what kind of things they're good/bad at.

Markov chains are super simple iterated next-word prediction model, and can be explained in 15 minutes. It's a great way to explain LLMs to laypeople.

Re: Markov Chains are the Original Language Models

#53

Post by Sohcahtoa82 »

adamnemecek wrote: post #17
bongodongobob wrote: post #14

Earlier quoted context omitted.

I've never seen a Markov chain do anything like GPT4. I'm not sure how you can say with a straight face they are basically the same.

No one has spent 100M on training Markov chains.

The trick with Markov chains is that you don't need to.

Markov Chains are dead simple. There's not really a "training" as much as it's simply reading data and collecting statistics.

They're so simple that you can probably build one nearly as fast as you can read the training data.

Re: Markov Chains are the Original Language Models

#54

Post by dankotanko1599 »

__mharrison__ wrote: post #28

Markov chains are fun. I often use them when teaching a Python fundamentals course. You can create an implementation in around 100 lines of code that explores many features of the language: classes, functions, loops, dictionaries, and lists. Then, you can augment with tests, a command line app, typing, etc.

How can I take your course?

Re: Markov Chains are the Original Language Models

#55

Post by BlueTemplar »

mewpmewp2 wrote: post #41
thwarted wrote: post #23

Earlier quoted context omitted.

A LLM is a Markov chain with billions of associations and weights. A Makov chain is an LLM of maybe a few dozen associations and weights (so an LM, without the first L). The difference is in the data structure and the size of the atoms/n-grams. The data structure Markov chain implementations use is not efficient for billions of parameters, either in storage or in processing. But the idea is the same: give a likely ne…

Idea behind complexity can be very simple, but at scale work yield in very different results. To compare Markov Chain with an LLM is kind of like to compare a single cell organism to a human being because we both are based on cells.

As already mentioned, Markov chains were already used in practice as (poor) Turing tests (IMHO working best in art projects, and sadly, spam).

Sure, today's LLMs blow them out of the water, but the difference was much less striking with neural networks even as late as 2010.

Re: Markov Chains are the Original Language Models

#56

Post by blt »

Sure, LLMs are like 1024-gram Markov chains (or whatever the context limit is). But there are problems: 1) the transition matrix is far too huge to represent, and 2) it treats all pairs of 1024-grams as completely different, even if the first 1023 words are the same.

Function approximation solves both issues, and the Transformer is the best function clas we've found so far.

Re: Markov Chains are the Original Language Models

#57

Post by candiodari »

marcusb wrote: post #44
thwarted wrote: post #23

Earlier quoted context omitted.

A LLM is a Markov chain with billions of associations and weights. A Makov chain is an LLM of maybe a few dozen associations and weights (so an LM, without the first L). The difference is in the data structure and the size of the atoms/n-grams. The data structure Markov chain implementations use is not efficient for billions of parameters, either in storage or in processing. But the idea is the same: give a likely ne…

“A human brain is just like a dog’s brain, only with more neural pathways.” True, perhaps, but largely pointless: at some point neural complexity results in a difference of kind, not of degree. I’d argue the same is true of LLMs vs simpler models like Markov chains.

But a human brain is also just like an elephant or whale's brain except with 1/5th of the neural pathways at best, probably less.

There's a qualitative difference too.

Re: Markov Chains are the Original Language Models

#58

Post by theGnuMe »

CamperBob2 wrote: post #42
thwarted wrote: post #23

Earlier quoted context omitted.

A LLM is a Markov chain with billions of associations and weights. A Makov chain is an LLM of maybe a few dozen associations and weights (so an LM, without the first L). The difference is in the data structure and the size of the atoms/n-grams. The data structure Markov chain implementations use is not efficient for billions of parameters, either in storage or in processing. But the idea is the same: give a likely ne…

It's not just window size. It's the difference between syntax and semantics. A Markov model, by definition, works only with literal token histories. It can't participate meaningfully in a conversation unless the user happens to employ token sequences that the model has seen before (ideally multiple times.) An LLM can explain why it's not just a Markov model, but the converse isn't true. Now, if you were to add high-d…

>high-dimensional latent-space embedding to a Markov model

That's what we call a hidden Markov model.

>There's a school of thought that says that lossy compression doesn't just require intelligence, it is intelligence, and LLMs can be seen as an example of that equivalence.

SVD is used to implement lossy compression as does JPEG encoding... these algorithms are in no way intelligent.

Re: Markov Chains are the Original Language Models

#59

Post by candiodari »

causalmodels wrote: post #40
thwarted wrote: post #23

Earlier quoted context omitted.

A LLM is a Markov chain with billions of associations and weights. A Makov chain is an LLM of maybe a few dozen associations and weights (so an LM, without the first L). The difference is in the data structure and the size of the atoms/n-grams. The data structure Markov chain implementations use is not efficient for billions of parameters, either in storage or in processing. But the idea is the same: give a likely ne…

An LLM is a Markov chain in the same sense that a cat is a tiger, technically true but it misses the qualia.

It's not. There's fundamental architectural differences that couldn't be bigger.

A better comparison would be that it's like a windup toy versus a group of humans moving an entire civilization. They both move along a distance, but just listing the systems that the human group has that the windup toy doesn't is too long to fit on a page.

Re: Markov Chains are the Original Language Models

#60

Post by theGnuMe »

hiddencost wrote: post #25
tanepiper wrote: post #8

This is what I've been saying to people, LLMs aren't much smarter than a Markov Chain - a lot of twitter bots and earlier chat agents were driven by them. What I do see though is the infrastructure around LLMs making a difference.

The whole point of Transformers is that they broke the Markov assumption (i.e., that the next token probability is strictly conditioned on a window of N preceding tokens).

Look into variable order Markov models.
Post reply on HN