Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

I don't know how you get here from “predict the next word”

grumpy-economist.com

Post reply on HN
Print view View original

191–200 of 275 posts

Re: I don't know how you get here from “predict the next word”

#191

Post by grey-area »

karamanolev wrote: post #136
grey-area wrote: post #122

Earlier quoted context omitted.

They generate text based on quite a large context, including hidden prompts we don’t see and their weights are distorted heavily by training. So I think there’s a lot more than a simple probability of word x coming next. That makes ‘predict next word’ a reductive summary IMO. I do not personally feel it resembles thinking or reasoning though and really object to that framing because it is misleading many people.

> their weights are distorted heavily by training What does that even mean? Their weights are essentially created by training. There aren't some magic golden weights that are then distorted.

I may be using the wrong terms, my impression was:

1. Weights in the model are created by ingesting the corpus

2. Techniques like reinforcement learning, alignment etc are used to adjust those weights before model release

3. The model is used and more context injected which then affects which words it will choose, though it is still heavily biased by the corpus and training.

That could be way off base though, I'd welcome correction on that.

The point I was trying to make though was that they do more than predict next word based on just one set of data. Their weights can encode entire passages of source material in the training data (https://arxiv.org/abs/2505.12546), including books, programs. This is why they are so effective at generating code snippets.

Also text injected at the last stage during use has far less weight than most people assume (e.g. https://georggrab.net/content/opus46retrieval.html) and is not read and understood IMO.

There are a lot of inputs nowadays and a lot of stages to training. So while I don't think they are intelligent I think it is reductive to call them next token predictors or similar. Not sure what the best name for them is, but they are neither next word predictors nor intelligent agents.

Re: I don't know how you get here from “predict the next word”

#192

Post by grey-area »

jbotz wrote: post #178
shakna wrote: post #99

Earlier quoted context omitted.

Probably worth remembering that ELIZA passed Turing tests, and was the definition of shallow prediction.

ELIZA absolutely did not ever pass anything resembling a real Turing test. A real Turing test is adversarial, the interrogator knows the testees are trying to fool him.

Unfortunately I'm not sure the Turing test posited a minimal level of intelligence for the human testers. As we have found with LLMs, humans are rather easy to fool.

Re: I don't know how you get here from “predict the next word”

#193

Post by apexalpha »

sasjaws wrote: post #93

A while ago i did the nanogpt tutorial, i went through some math with pen and paper and noticed the loss function for 'predict the next token' and 'predict the next 2 tokens' (or n tokens) is identical. That was a bit of a shock to me so wanted to share this thought. Basically i think its not unreasonable to say llms are trained to predict the next book instead of single token. Hope this is usefull to someone.

Are you referring to this one?: https://github.com/karpathy/build-nanogpt

Re: I don't know how you get here from “predict the next word”

#194

Post by fc417fc802 »

qsera wrote: post #176
fc417fc802 wrote: post #168

Earlier quoted context omitted.

The distinction you're making reads like substance dualism to me. Are you able to provide a clear and objective metric for assessing "understanding"? If not then you're just handwaving an effectively meaningless semantic distinction.

>objective metric for assessing "understanding" It should involve consciousness. You would not call an AI reacting to red color as "seeing" red. Same thing.

And where is this objective metric for consciousness? Last I checked we didn't even have a sensible definition for it.

It seems to me you're just kicking the can.

Setting that issue aside. While I certainly don't believe LLMs to be conscious (an entirely subjective and arbitrary take on my part I admit) I don't see any reason that concepts such as "intelligence" and "understanding" should require it. When considering how we apply those terms to humans it seems to me they are results based and highly contextual (ie largely arbitrary).

Re: I don't know how you get here from “predict the next word”

#195

Post by grey-area »

red75prime wrote: post #172
grey-area wrote: post #70

Earlier quoted context omitted.

Given their failure on novel logic problems, generation of meaningless text, tendency to do things like delete tests and incompetence at simple mathematics, it seems very unlikely they have built any sort of world model. It’s remarkable how competent they are given the way they work. Predict the next word is a terrible summary of what these machines do though, they certainly do more than that, but there are significa…

> there are significant limitations Where can we read about those significant limitations?

Well here's some:

Confabulation/Hallucination - https://github.com/lechmazur/confabulations

Failure to read context - https://georggrab.net/content/opus46retrieval.html

Deleting tests to make them pass - https://www.linkedin.com/posts/jasongorman_and-after-it-did-...

Going rogue and deleting data - https://x.com/jasonlk/status/1946069562723897802

Agent security nightmares because they are not in fact intelligent assistants - https://x.com/theonejvo/status/2015401219746128322

Failure to read or generate structured data - https://support.google.com/gemini/thread/390981629/llm-ignor...

There are many, many examples, mostly caused by people thinking LLMs are intelligent and reasoning and giving them too much power (e.g. treating them as agents, not text generators). I'm sure they're all fixed in whatever new version came out this week though.

Re: I don't know how you get here from “predict the next word”

#196

Post by grey-area »

smallpipe wrote: post #170
jjmarr wrote: post #81

Earlier quoted context omitted.

I created a code review pipeline at work with a similar tradeoff and we found the cost is worth it. Time is a non-issue. We could run Claude on our code and call it a day, but we have hundreds of style, safety, etc rules on a very large C++ codebase with intricate behaviour (cooperative multitasking be fun). So we run dozens of parallel CLI agents that can review the code in excruciating detail. This has completely r…

> This has completely replaced human code review for anything that isn't functional correctness Isn’t functional correctness pretty much the only thing that matters though?

Well no, style is important too for humans when they read a codebase, so the LLMs the parent is running clearly have some value for them.

They're not claiming LLMs solved every problem, just that they made life easier by taking care of busywork that humans would otherwise be doing. I think personally this is quite a good use for them - offering suggestions on PRs say, as long as humans still review them as well.

Re: I don't know how you get here from “predict the next word”

#197

Post by js8 »

phreeza wrote: post #48
auraham wrote: post #15

Earlier quoted context omitted.

I highly recommend Build a large language model from scratch [1] by Sebastian Raschka. It provides a clear explanation of the building blocks used in the first versions of ChatGPT (GPT 2 if I recall correctly). The output of the model is a huge vector of n elements, where n is the number of tokens in the vocabulary. We use that huge vector as a probability distribution to sample the next token given an input sequence…

But this is missing exactly the gap which OP seems to have, which is going from a next token predictor (a language model in the classical sense) to an instruction finetuned, RLHF-ed and "harnessed" tool?

The book has a sequel https://www.manning.com/books/build-a-reasoning-model-from-s...

It will give you an answer to the extent anybody can.

Re: I don't know how you get here from “predict the next word”

#198

Post by karamanolev »

grey-area wrote: post #191
karamanolev wrote: post #136

Earlier quoted context omitted.

> their weights are distorted heavily by training What does that even mean? Their weights are essentially created by training. There aren't some magic golden weights that are then distorted.

I may be using the wrong terms, my impression was: 1. Weights in the model are created by ingesting the corpus 2. Techniques like reinforcement learning, alignment etc are used to adjust those weights before model release 3. The model is used and more context injected which then affects which words it will choose, though it is still heavily biased by the corpus and training. That could be way off base though, I'd wel…

That extended explanation is more accurate, yes. I'd call your points 1 and 2 both training under the definition "anything that adjusts model weights is training". There are multiple stages and types of training. Right now AFAIK most (all) architectures then fix the weights and you have non-weight-affecting steps like the system prompt, context, etc.

You're right that the weights can enable the model to memorize training data.

Re: I don't know how you get here from “predict the next word”

#199

Post by withinboredom »

Kim_Bruning wrote: post #189
withinboredom wrote: post #164

Earlier quoted context omitted.

But the ideas are not 'new'. A benchmark that I use to tell me if an AI is overfitted is to present the AI with a recent paper (especially one like a paxos variant) and have it build that. If it writes general paxos instead of what the paper specified, its overfitted. Claude 4.5: not overfitted too much -- does the right thing 6/10 times. Claude 4.6: overfitted -- does the right thing 2/10 times. OpenAI 5.3: overfitt…

Could also be that the model has stronger priors wrt Paxos (and thus has Opinions on what good Paxos should look like) At any rate, with an assembler, you end up with a lot of random letter-salad mnemonics with odd use cases, so that is very likely to tokenize in interesting ways at the very least.

I was just using paxos as an example. Any paper will do.

Re: I don't know how you get here from “predict the next word”

#200

Post by qsera »

fc417fc802 wrote: post #194
qsera wrote: post #176

Earlier quoted context omitted.

>objective metric for assessing "understanding" It should involve consciousness. You would not call an AI reacting to red color as "seeing" red. Same thing.

And where is this objective metric for consciousness? Last I checked we didn't even have a sensible definition for it. It seems to me you're just kicking the can. Setting that issue aside. While I certainly don't believe LLMs to be conscious (an entirely subjective and arbitrary take on my part I admit) I don't see any reason that concepts such as "intelligence" and "understanding" should require it. When considering…

>humans it seems to me they are results based and highly contextual (ie largely arbitrary).

Is that right? It seems that we generally say that "the computer is programmed to do", instead of "the computer understand" or "the computer knows", even if the programmed computer can produce the same result as a human who does it.

Post reply on HN