Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Markov Chains are the Original Language Models

elijahpotter.dev

Post reply on HN
Print view View original

111–120 of 177 posts

Re: Markov Chains are the Original Language Models

#111

Post by SilasX »

svachalek wrote: post #102
SilasX wrote: post #94

Earlier quoted context omitted.

Did read the part about king - man + woman = queen? How is that a next-token predictor?

With the caveat that I'm not an LLM expert here but have read up on some of this... What's basically going on is that the LLM has "read" vast amounts of text and classified words in all kinds of dimensions. As it goes from word to word it's predicting based on all of those dimensions, rather than the Markov chain's simple probabilities. So it knows "king" has high values of masculinity and authority, for example, and…

So you're saying that the word2vec king/queen result was the result of feeding the verbatim text "king+man-woman", and that text being continued as "queen"? I assumed that was more the result of doing math on the properties the model generated for the tokens king, man, queen, etc.

And in that case, why does "Write me a limerick " result in ChatGPT producing limericks, when vanishingly few of the limericks in the source text started that way, and vanishingly few of the commands to write a limerick were immediately followed with one?

Re: Markov Chains are the Original Language Models

#112

Post by thfuran »

skrebbel wrote: post #74
thfuran wrote: post #70

Earlier quoted context omitted.

A typical demonstration markov chain probably has a length of around 3. A typical recent LLM probably has more than three billion parameters. That's not precisely apppes to apples, but the LLM is certainly vastly more complicated.

The way you describe it, it doesn't seem much more complicated to me, from a “how does it work” perspective, just way bigger.

If the minimal representation of a model of the behavior is "way bigger", why are you disputing that it's more complicated? What's the difference?

Re: Markov Chains are the Original Language Models

#113

Post by marcus0x62 »

andoando wrote: post #93
marcusb wrote: post #44

Earlier quoted context omitted.

“A human brain is just like a dog’s brain, only with more neural pathways.” True, perhaps, but largely pointless: at some point neural complexity results in a difference of kind, not of degree. I’d argue the same is true of LLMs vs simpler models like Markov chains.

I think this vastly underplays animals intelligence though. There is so much focus on creating human level intelligence, but where is a robot that can learn to navigate the world like a dog or cat can?

It was an analogy about how stupid it is to say a markov chain is “just like an LLM.”

Re: Markov Chains are the Original Language Models

#114

Post by klyrs »

jjtheblunt wrote: post #68
lukev wrote: post #52

What's actually happening in a LLM is many orders of magnitude more complex than a Markov chain. However, I agree that they're an amazing pedagogical tool for the basic principles of how a LLM works, even to a non-technical audience. Many people try to "explain" LLMs starting with the principles of neural networks. This rarely works well: there are some significant conceptual leaps required. However, explaining that…

Genuine question: what do you mean by many orders of magnitude more complex?

I like your question, and I cannot answer it. But I have a benchmark: I can write a Markov chain "language model" in around 10-20 lines of Python, with zero external libraries -- with tokenization and "training" on a text file, and generating novel output. I wrote it in several minutes and didn't bother to save it.

I'm curious how much time & code it would take to implement this LLM stuff at a similar level of quality and performance.

Re: Markov Chains are the Original Language Models

#115

Post by x-complexity »

devmor wrote: post #3

Markov chains were one of the coolest discoveries of my programming career and I spent years using them to make forum and social media bots trained on people’s post histories for them. I think that experience is part of why I’ve been generally unimpressed with a lot of LLM hype. Like yeah, it’s cool and definitely more useful than a markov chain - but for the amount of resources that went into it I’d expect the gap t…

> I think that experience is part of why I’ve been generally unimpressed with a lot of LLM hype.

There are fundamental scale & architectural differences that make LLMs different from Markov Chains: It's like saying that you're not impressed by indoor plumbing because you have experience carrying your water from a well, and that both do the same thing - transporting water to your home.

In both cases, this line of logic ignores the improvements made to make such a thing remotely possible, and the difference in relative usefulness that can be gained from LLMs in comparison to Markov Chains.

Re: Markov Chains are the Original Language Models

#116

Post by aoeusnth1 »

pyinstallwoes wrote: post #81
rwmj wrote: post #75

Earlier quoted context omitted.

Number of parameters is not the difference. A Markov chain can easily be a multi-dimensional matrix with millions of entries. The significant difference is that a length 3 Markov chain can only ever find connections between 3 adjacent symbols (words, usually). LLMs seem to be able to find and connect abstract concepts at a very long and variable distances in the input. Nevertheless I agree with the premise of the pos…

Seemingly? Is there not a direct technical reason to compare?

Reproduction context length is a standard benchmark.

Re: Markov Chains are the Original Language Models

#117

Post by soraki_soladead »

klyrs wrote: post #114
jjtheblunt wrote: post #68

Earlier quoted context omitted.

Genuine question: what do you mean by many orders of magnitude more complex?

I like your question, and I cannot answer it. But I have a benchmark: I can write a Markov chain "language model" in around 10-20 lines of Python, with zero external libraries -- with tokenization and "training" on a text file, and generating novel output. I wrote it in several minutes and didn't bother to save it. I'm curious how much time & code it would take to implement this LLM stuff at a similar level of qualit…

FLOPs by perplexity by samples is an interesting way to compare this family of models.

Re: Markov Chains are the Original Language Models

#118

Post by islewis »

I resource I found invaluable in demystifying LLM's was Andrej Karpathy's Youtube series. In particular, his "Makemore" video built a basic LLM in the same vein as OP's.

It's long (2h), but well worth the time if anyone is curious about the technical workings of LLMs.

https://www.youtube.com/watch?v=PaCmpygFfXo&list=PLAqhIrjkxb...

Re: Markov Chains are the Original Language Models

#119

Post by bormaj »

unoti wrote: post #38

It's true that Markov chains are very limited in their capabilities. But one thing I love about them is that they are one of the simplest and most intuitive ways to write code that *learns* from input data. If you're never written something that *learns*, try it out! Here's a very primitive one I wrote recently to explain the basic idea and explains it along the way. https://github.com/unoti/markov-basics/blob/main/m…

Thank you for sharing this! The implementation and examples were super intuitive.
Post reply on HN