Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

I don't know how you get here from “predict the next word”

grumpy-economist.com

Post reply on HN
Print view View original

171–180 of 275 posts

Re: I don't know how you get here from “predict the next word”

#171

Post by keybored »

> On reflection I have started to worry again. In 10 to 20 years nobody will read anything any more, they just will read LLM digests. So, the single most important task of a writer starting right now is to get your efforts wired in to the LLMs. Nothing you write will matter if it is not quickly adopted to the training dataset. As the art of pushing your results to the top of the google search was the 1990s game, getting your ideas into the LLMs is today’s. Refine is no different. It’s so good, everyone will use it. So whether refine and its cousins take a FTPL or new Keynesian view in evaluating papers is now all determining for where the consensus of the profession goes.

I expected a bit more cynicism and less merrily going with the downward spiral flow from a “grumpy” blogger.

Oh, but it’s a grumpy economist.

Re: I don't know how you get here from “predict the next word”

#172

Post by red75prime »

grey-area wrote: post #70
ChaitanyaSai wrote: post #43

The whole next word thing is interesting isn't it. I like to see it with Dennett's "Competence and comprehension" lens. You can predict the next word competently with shallow understanding. But you could also do it well with understanding or comprehension of the full picture. A mental model that allows you to predict better. Are the AIs stumbling into these mental models? Seems like it. However, because these are suc…

Given their failure on novel logic problems, generation of meaningless text, tendency to do things like delete tests and incompetence at simple mathematics, it seems very unlikely they have built any sort of world model. It’s remarkable how competent they are given the way they work. Predict the next word is a terrible summary of what these machines do though, they certainly do more than that, but there are significa…

> there are significant limitations

Where can we read about those significant limitations?

Re: I don't know how you get here from “predict the next word”

#173

Post by WithinReason »

selridge wrote: post #69
anon7725 wrote: post #54

Earlier quoted context omitted.

“Represented in the training data” does not mean “represented as a whole in the training data”. If A and B are separately in the training data, the model can provide a result when A and B occur in the input because the model has made a connection between A and B in the latent space.

Yes. I’m saying that “it’s just in the training data” is a cognitive containment of these models which is incomplete. You can insist that’s what’s happening, but you’ll be left unable to explain what’s going on beyond truisms.

It's called "generalization":

https://en.wikipedia.org/wiki/Generalization_(learning)

Re: I don't know how you get here from “predict the next word”

#174

Post by red75prime »

steve1977 wrote: post #106
mzhaase wrote: post #100

Earlier quoted context omitted.

It always occurred to me that LLMs may be like the language center of the brain. And there should be a "whole damn rest of the brain" behind it to steer it. LLMs miss very important concepts, like the concept of a fact. There is no "true", just consensus text on the internet given a certain context. Like that study recently where LLMs gave wrong info if there was the biography of a poor person in the context.

I think much along the same lines. LLMs are probably even just a part of the language center. And of course they also miss things like embodiment, mirror neurons etc. If an LLM makes a mistake, it will tell you it is sorry. But does it really feel sorry?

> But does it really feel sorry?

And what does it mean to feel sorry? Beyond fallible and imprecise human introspective notion of "sorry", that is. A definition that can span species and computing substrates. A deanthropomorphized definition of "sorry", so to speak.

Re: I don't know how you get here from “predict the next word”

#175

Post by croon »

sasjaws wrote: post #93

A while ago i did the nanogpt tutorial, i went through some math with pen and paper and noticed the loss function for 'predict the next token' and 'predict the next 2 tokens' (or n tokens) is identical. That was a bit of a shock to me so wanted to share this thought. Basically i think its not unreasonable to say llms are trained to predict the next book instead of single token. Hope this is usefull to someone.

Isn't that why noise was introduced (seed rolling/temperature/high p/low p/etc)? I mean it is still deterministic given the same parameters.

But this might be misleadingly interpreted as an LLM having "thought out an answer" before generating tokens, which is an incorrect conclusion.

Not suggesting you did.

Re: I don't know how you get here from “predict the next word”

#176

Post by qsera »

fc417fc802 wrote: post #168
qsera wrote: post #118

Earlier quoted context omitted.

Yea, semantics is important. It is not "understanding" any more than a microphone+ADC is hearing.

The distinction you're making reads like substance dualism to me. Are you able to provide a clear and objective metric for assessing "understanding"? If not then you're just handwaving an effectively meaningless semantic distinction.

>objective metric for assessing "understanding"

It should involve consciousness. You would not call an AI reacting to red color as "seeing" red. Same thing.

Re: I don't know how you get here from “predict the next word”

#177

Post by qsera »

echelon wrote: post #141
qsera wrote: post #132

Earlier quoted context omitted.

> money-making companies You mean, money sucking companies, right? >You're removed from orders of magnitude in upside potential if you have to wait for the public markets. because that won't work. That is why!

> You mean, money sucking companies, right? Is that what you (and all people) are in your job function? A money suck? Do you ever buy anything for food, shelter, and clothing? Do you have hobbies? Capitalism means we don't have to all be hunter-gatherers, and I'm pretty keen on that trade. > because that won't work. That is why! This is the forum for a venture capital firm. A lot of the folks here build things with t…

I get paid for what I do. Not for what I promise to do.

Re: I don't know how you get here from “predict the next word”

#178

Post by jbotz »

shakna wrote: post #99
grey-area wrote: post #70

Earlier quoted context omitted.

Given their failure on novel logic problems, generation of meaningless text, tendency to do things like delete tests and incompetence at simple mathematics, it seems very unlikely they have built any sort of world model. It’s remarkable how competent they are given the way they work. Predict the next word is a terrible summary of what these machines do though, they certainly do more than that, but there are significa…

Probably worth remembering that ELIZA passed Turing tests, and was the definition of shallow prediction.

ELIZA absolutely did not ever pass anything resembling a real Turing test. A real Turing test is adversarial, the interrogator knows the testees are trying to fool him.

Re: I don't know how you get here from “predict the next word”

#179

Post by ainch »

wavemode wrote: post #41
selridge wrote: post #34

Earlier quoted context omitted.

>You're implicitly assuming that what you asked the LLM to do is unrepresented in the training data. This is just as stuck in a moment in time as "they only do next word prediction" What does this even mean anymore? Are we supposed to believe that a review of this paper that wasn't written when that model (It's putatively not an "LLM", but IDK enough about it to be pushy there) was trained? Does that even make sense?…

Statistical models generalize. If you train a model that f(x) = 5 and f(x+1) = 6, the number 7 doesn't have to exist in the training data for the model to give you a correct answer for f(x+2) Similarly, if there are millions of academic papers and thousands of peer reviews in the training data, a review of this exact paper doesn't need to be in there for the LLM to write something convincing. (I say "convincing" rath…

Sorry but this is famously not true! There is no guarantee that statistical models generalise. In your example, whether or not your model generalises depends entirely on what f(x) you use - depending on the complexity of your function class f(x+2) could be 7, 8, or -500.

One of the surprises of deep learning is that it can, sometimes, defy prior statistical learning theory to generalise, but this is still poorly understood. Concepts like grokking, double descent, and the implicit bias of gradient descent are driving a lot of new research into the underlying dynamics of deep learning. But I'd say it is pretty ahistoric to claim that this is obvious or trivial - decades of work studied "overfitting" and related problems where statistical models fail to generalise or even interpolate within the support of their training data.

Re: I don't know how you get here from “predict the next word”

#180

Post by throw310822 »

croon wrote: post #175
sasjaws wrote: post #93

A while ago i did the nanogpt tutorial, i went through some math with pen and paper and noticed the loss function for 'predict the next token' and 'predict the next 2 tokens' (or n tokens) is identical. That was a bit of a shock to me so wanted to share this thought. Basically i think its not unreasonable to say llms are trained to predict the next book instead of single token. Hope this is usefull to someone.

Isn't that why noise was introduced (seed rolling/temperature/high p/low p/etc)? I mean it is still deterministic given the same parameters. But this might be misleadingly interpreted as an LLM having "thought out an answer" before generating tokens, which is an incorrect conclusion. Not suggesting you did.

> this might be misleadingly interpreted as an LLM having "thought out an answer"

I'm convinced that that is exactly what happens. Anthropic confirms it:

"Claude will plan what it will say many words ahead, and write to get to that destination. We show this in the realm of poetry, where it thinks of possible rhyming words in advance and writes the next line to get there. This is powerful evidence that even though models are trained to output one word at a time, they may think on much longer horizons to do so."

https://www.anthropic.com/research/tracing-thoughts-language...

Post reply on HN