Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Markov Chains are the Original Language Models

elijahpotter.dev

Post reply on HN
Print view View original

21–30 of 177 posts

Re: Markov Chains are the Original Language Models

#21

Post by deepsquirrelnet »

> This a scratch text area to test the autocomplete in another moment down would be four thousand miles i wish you know please ma am is this new zealand or conversations in another moment down stairs how funny it

^ to me it’s really a stretch that someone could read that and think “markov chains are practically an llm!”

They’re pretty limited, and everyone working on building LLMs knows about them and has probably used them at some point.

They both generate tokens/words on probabilities, but it’s hard to understate how much simpler of a probability model a Markov chain is compared to an llm.

You’ve stated that markov chains have a limited use as a simple chat bot. How do you improve an mc so that it can perform other tasks, like classification, summarization, question answering, complex text parsing, text matching, etc, etc?

The language modeling done in llms has an open pathway to scaling and improving on this statistical model, and it’s incredible to me that this is still opaque to people. I don’t believe that llms are a path to intelligence, but I can certainly recognize that they’re going to emerge as a sophisticated and very useful productivity tool.

Re: Markov Chains are the Original Language Models

#22

Post by cmrdporcupine »

Someone wrote a Markov chaining bot back on LambdaMOO in the early 90s, that would pretend to participate in conversation in the room, and I remember being blown away by it. Especially in the context of the limited computation power on that platform and the machine it was hosted on.

Re: Markov Chains are the Original Language Models

#23

Post by thwarted »

bongodongobob wrote: post #14
tanepiper wrote: post #8

This is what I've been saying to people, LLMs aren't much smarter than a Markov Chain - a lot of twitter bots and earlier chat agents were driven by them. What I do see though is the infrastructure around LLMs making a difference.

I've never seen a Markov chain do anything like GPT4. I'm not sure how you can say with a straight face they are basically the same.

A LLM is a Markov chain with billions of associations and weights. A Makov chain is an LLM of maybe a few dozen associations and weights (so an LM, without the first L).

The difference is in the data structure and the size of the atoms/n-grams. The data structure Markov chain implementations use is not efficient for billions of parameters, either in storage or in processing. But the idea is the same: give a likely next token given the last n tokens. The value for n is a narrow window for Markov chains and an extremely wide window for LLM. LLM are able to maintain massive amounts of state compared to a Markov chain implementation.

Re: Markov Chains are the Original Language Models

#24

Post by chpatrick »

tanepiper wrote: post #8

This is what I've been saying to people, LLMs aren't much smarter than a Markov Chain - a lot of twitter bots and earlier chat agents were driven by them. What I do see though is the infrastructure around LLMs making a difference.

"humans aren't much smarter than goldfish"

Re: Markov Chains are the Original Language Models

#25

Post by hiddencost »

tanepiper wrote: post #8

This is what I've been saying to people, LLMs aren't much smarter than a Markov Chain - a lot of twitter bots and earlier chat agents were driven by them. What I do see though is the infrastructure around LLMs making a difference.

The whole point of Transformers is that they broke the Markov assumption (i.e., that the next token probability is strictly conditioned on a window of N preceding tokens).

Re: Markov Chains are the Original Language Models

#26

Post by hiddencost »

adamnemecek wrote: post #17
bongodongobob wrote: post #14

Earlier quoted context omitted.

I've never seen a Markov chain do anything like GPT4. I'm not sure how you can say with a straight face they are basically the same.

No one has spent 100M on training Markov chains.

Google did it. Amazon did it. Plenty of others did. What do you think they were doing before recurrent neural networks?

Re: Markov Chains are the Original Language Models

#27

Post by pclmulqdq »

hiddencost wrote: post #25
tanepiper wrote: post #8

This is what I've been saying to people, LLMs aren't much smarter than a Markov Chain - a lot of twitter bots and earlier chat agents were driven by them. What I do see though is the infrastructure around LLMs making a difference.

The whole point of Transformers is that they broke the Markov assumption (i.e., that the next token probability is strictly conditioned on a window of N preceding tokens).

That's not actually true, they still have a fixed history window. The idea that transformers capture through the attention mechanism is that not all past tokens are created equal, and that the importance of tokens in that history window depends on what they are.

Re: Markov Chains are the Original Language Models

#28

Post by __mharrison__ »

Markov chains are fun. I often use them when teaching a Python fundamentals course. You can create an implementation in around 100 lines of code that explores many features of the language: classes, functions, loops, dictionaries, and lists. Then, you can augment with tests, a command line app, typing, etc.

Re: Markov Chains are the Original Language Models

#30

Post by CamperBob2 »

tanepiper wrote: post #8

This is what I've been saying to people, LLMs aren't much smarter than a Markov Chain - a lot of twitter bots and earlier chat agents were driven by them. What I do see though is the infrastructure around LLMs making a difference.

This is what I've been saying to people, LLMs aren't much smarter than a Markov Chain

And that's why people have been nodding, smiling politely and reaching behind their back for the doorknob.

Post reply on HN