Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Markov Chains are the Original Language Models

elijahpotter.dev

Post reply on HN
Print view View original

101–110 of 177 posts

Re: Markov Chains are the Original Language Models

#101

Post by famouswaffles »

lukev wrote: post #90
wongarsu wrote: post #67

Earlier quoted context omitted.

They are an especially useful tool right now, that might become less valuable as we get better at building LLMs. In principle the inner working of an LLM can be anything from a Markov-chain-like predictor to a beyond-human intelligence. Token prediction is the input/output format we chose, but you could communicate with a human in the same format and the human would show human-level intelligence. What makes Markov ch…

> In principle the inner working of an LLM can be anything from a Markov-chain-like predictor to a beyond-human intelligence. I'm afraid I have to disagree. Next-token prediction isn't just the interface we use for LLMs, it is fundamentally what they are , to the very core. The training and loss function of the foundation models are completely oriented towards next-token accuracy. Reasonable people can disagree about…

>The training and loss function of the foundation models are completely oriented towards next-token accuracy.

This doesn't mean anything.

Loss function and training only concern themselves with the result of the prediction. The in-between, the computation, training does not care except as a means to an end.

It's not Input A > Output B. It's Input A > Computation > Output B.

That Computation could quite literally be anything. And no, we do not automatically know what this computation might represent or if it even represents anything that would be understandable by us.

If you train a Transformer meticulously on predicting the token that is the result of an addition, you might hope the computation it learns is some algorithm for addition but you wouldn't actually know until you attempted to and successfully probed the model.

Re: Markov Chains are the Original Language Models

#102

Post by svachalek »

SilasX wrote: post #94
benreesman wrote: post #89

Earlier quoted context omitted.

Predicting subsequent text is pretty much exactly what they do. Lots of very cool engineering that’s a real feat, but at its core it’s argmax(P(token|token,corpus)): https://github.com/facebookresearch/llama/blob/main/llama/ge... The engineering feats are up there with anything, but it’s a next token predictor.

Did read the part about king - man + woman = queen? How is that a next-token predictor?

With the caveat that I'm not an LLM expert here but have read up on some of this...

What's basically going on is that the LLM has "read" vast amounts of text and classified words in all kinds of dimensions. As it goes from word to word it's predicting based on all of those dimensions, rather than the Markov chain's simple probabilities. So it knows "king" has high values of masculinity and authority, for example, and "man" has high values of masculinity without necessary the authority. Likewise for queen and woman. This also works for connective words like plus, minus, equals, etc. This leads the LLM to judge correctly that the most plausible word to continue that equation is "queen".

With enough dimensions and enough context (preceding words to include in the calculation) you get results that look like reasoning and intelligence -- and although a lot of people have started arguing that we need to define reasoning and intelligence in a way that excludes this, I'm not so sure of that. It seems quite possible that what goes on in our own heads is not so far from this with a few extra steps.

Re: Markov Chains are the Original Language Models

#104

Post by BlueTemplar »

bbor wrote: post #76
BlueTemplar wrote: post #55

Earlier quoted context omitted.

As already mentioned, Markov chains were already used in practice as (poor) Turing tests (IMHO working best in art projects, and sadly, spam). Sure, today's LLMs blow them out of the water, but the difference was much less striking with neural networks even as late as 2010.

That’s interesting you put the date at 2010! When I learned machine learning in ~2016-2018, markov chains were THE fundamental problem tool of NLP still. Neural networks were all the rage ofc, but still… could you tell me what tech/change you’re thinking of? For example, I remember /r/SubredditSimulator blowing my mind, and I’m pretty sure that was markov chains

Yeah, my bad, I took a guess here, from what I remember about when neural networks became the hot buzzword (again, after the 1970's Perceptron ?) - sounds like I missed, since the transition was slower than I thought ?

P.S.: "2010" was probably from this great Ars Technica article :

https://arstechnica.com/gaming/2011/01/skynet-meets-the-swar...

It's not clear to me whether neural networks were already involved (though there's already some self-learning there), you can see a few Ars and YouTube comments mentioning them, though "genetic algorithms" seems to also have been a popular buzzword ?

P.P.S.: This is about StarCraft 1 (ONE), not to be confused with DeepMind's Alpha Star wiping the floor with a pro StarCraft 2 (TWO) player in 2018.

Re: Markov Chains are the Original Language Models

#105

Post by rany_ »

lormayna wrote: post #103

In 2016 I created a Twitter bot that replicated a political activist via a Markov chain model. It was impressive how many people liked those post, also a middle level politician.

Can you link to that Twitter bot? I'm just curious about the quality of the posts it generated.

Re: Markov Chains are the Original Language Models

#106

Post by benreesman »

SilasX wrote: post #94
benreesman wrote: post #89

Earlier quoted context omitted.

Predicting subsequent text is pretty much exactly what they do. Lots of very cool engineering that’s a real feat, but at its core it’s argmax(P(token|token,corpus)): https://github.com/facebookresearch/llama/blob/main/llama/ge... The engineering feats are up there with anything, but it’s a next token predictor.

Did read the part about king - man + woman = queen? How is that a next-token predictor?

word2vec is a very cool result, but suggestive at best for how a modern LLM works. in fact the king + country - queen + capital vector Mary you’re referring to is pretty much a direct consequence of excluding nonlinear ties from the SGD-driven matrix factorization process.

not everything done via SGD is a ReLU :)

Re: Markov Chains are the Original Language Models

#107

Post by esafak »

PheonixPharts wrote: post #15
bongodongobob wrote: post #14

Earlier quoted context omitted.

I've never seen a Markov chain do anything like GPT4. I'm not sure how you can say with a straight face they are basically the same.

LLMs are Markov chains in latent space , it's the latent representation that give them their power, but ultimately there's not as much difference as one would suspect.

They're different because Markov models are stateless whereas LLMs are stateful.

https://en.wikipedia.org/wiki/Markov_property

Re: Markov Chains are the Original Language Models

#108

Post by esafak »

adamnemecek wrote: post #17
bongodongobob wrote: post #14

Earlier quoted context omitted.

I've never seen a Markov chain do anything like GPT4. I'm not sure how you can say with a straight face they are basically the same.

No one has spent 100M on training Markov chains.

It would make no sense; they are not powerful models worthy of the spend.

Re: Markov Chains are the Original Language Models

#109

Post by [deleted] »

pclmulqdq wrote: post #27
hiddencost wrote: post #25

Earlier quoted context omitted.

The whole point of Transformers is that they broke the Markov assumption (i.e., that the next token probability is strictly conditioned on a window of N preceding tokens).

That's not actually true, they still have a fixed history window. The idea that transformers capture through the attention mechanism is that not all past tokens are created equal, and that the importance of tokens in that history window depends on what they are.

[deleted]

Re: Markov Chains are the Original Language Models

#110

Post by benreesman »

tsimionescu wrote: post #97
SilasX wrote: post #94

Earlier quoted context omitted.

Did read the part about king - man + woman = queen? How is that a next-token predictor?

It works by computing that P(queen|"king-man+woman", corpus) > P( |"king-man+woman", corpus), i.e. it predicts that the most likely next token after that phrase, based on the entire training corpus and the loss function, is "queen". Now, how exactly the LLM is computing this prediction remains poorly understood.

word2vec is pretty much completely understood. there are maybe like some bounds that could maybe pushed lower but we know exactly how it works.
Post reply on HN