Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

I don't know how you get here from “predict the next word”

grumpy-economist.com

Post reply on HN
Print view View original

111–120 of 275 posts

Re: I don't know how you get here from “predict the next word”

#111

Post by basch »

qsera wrote: post #98
basch wrote: post #95

Earlier quoted context omitted.

It's honestly disheartening and a bit shocking how everyone has started repeating the predict the next syllable criticism. The language model predicts the next syllable by FIRST arriving in a point in space that represents UNDERSTANDING of the input language. This was true all the way back in 2017 at the time of Attention Is All You Need. Google had a beautiful explainer page of how transformers worked, which I am st…

>represents UNDERSTANDING of the input language. It does not have an understanding, it pattern matches the "idea shape" of words in the "idea space" of training data and calculates the "idea shape" that is likely to follow considering all the "idea shape" patterns in its training data. It mimics understanding. It feels mysterious to us because we cannot imagine the mapping of a corpus of text to this "idea space". It…

Semantics. Its a encoded position that represents meaning in a way that is useful and reusable. That is "understanding." It's a mathematical representation of grasp.

Re: I don't know how you get here from “predict the next word”

#112

Post by teekert »

I think this is a thing not often discussed here, but I too have this experience. An LLM can be fantastic if you write a 25-pager then later need to incorporate a lot of comments with sometimes conflicting arguments/viewpoints.

LLMs can be really good at "get all arguments against this", "Incorporated this view point in this text while making it more concise.", "Are these views actually contradicting or can I write it such that they align. Consider incentives".

If you know what you're doing and understand the matter deeply (and that is very important) you'll find that the LLM is sometimes better at wording what you actually mean, especially when not writing in your native language. Of course, you study the generated text, make small changes, make it yours, make sure you feel comfortable with it etc. But man can it get you over that "how am I going to write this down"-hump.

Also: "Make an executive summary" "Make more concise", are great. Often you need to de-linkedIn the text, or tell it to "not sound like an American waiter", and "be business-casual", "adopt style of rest of doc", etc. But it works wonders.

Re: I don't know how you get here from “predict the next word”

#113

Post by basch »

steve1977 wrote: post #108
basch wrote: post #95

Earlier quoted context omitted.

It's honestly disheartening and a bit shocking how everyone has started repeating the predict the next syllable criticism. The language model predicts the next syllable by FIRST arriving in a point in space that represents UNDERSTANDING of the input language. This was true all the way back in 2017 at the time of Attention Is All You Need. Google had a beautiful explainer page of how transformers worked, which I am st…

From what I understand, it's more like "input is 1, 3, 5, 7" so "output is likely to be 9". Understanding would be a bit generous of a term for that I guess, but that also depends on the definition of understanding.

Id really invite people to read the google blog post. https://research.google/blog/transformer-a-novel-neural-netw...

Google chose the word understanding.

Re: I don't know how you get here from “predict the next word”

#115

Post by steve1977 »

basch wrote: post #113
steve1977 wrote: post #108

Earlier quoted context omitted.

From what I understand, it's more like "input is 1, 3, 5, 7" so "output is likely to be 9". Understanding would be a bit generous of a term for that I guess, but that also depends on the definition of understanding.

Id really invite people to read the google blog post. https://research.google/blog/transformer-a-novel-neural-netw... Google chose the word understanding.

Thanks for the link, I will read it. But keep in mind that Google wants to sell us something.

Re: I don't know how you get here from “predict the next word”

#116

Post by aqua_coder »

qsera wrote: post #98
basch wrote: post #95

Earlier quoted context omitted.

It's honestly disheartening and a bit shocking how everyone has started repeating the predict the next syllable criticism. The language model predicts the next syllable by FIRST arriving in a point in space that represents UNDERSTANDING of the input language. This was true all the way back in 2017 at the time of Attention Is All You Need. Google had a beautiful explainer page of how transformers worked, which I am st…

>represents UNDERSTANDING of the input language. It does not have an understanding, it pattern matches the "idea shape" of words in the "idea space" of training data and calculates the "idea shape" that is likely to follow considering all the "idea shape" patterns in its training data. It mimics understanding. It feels mysterious to us because we cannot imagine the mapping of a corpus of text to this "idea space". It…

I am not knowledgeable on how transformer works but, what if, us humans just do the same thing in our minds as well ? What if our feeling of "understanding" is merely just the emotional response to a pattern matching as you just said?

Re: I don't know how you get here from “predict the next word”

#117

Post by basch »

steve1977 wrote: post #115
basch wrote: post #113

Earlier quoted context omitted.

Id really invite people to read the google blog post. https://research.google/blog/transformer-a-novel-neural-netw... Google chose the word understanding.

Thanks for the link, I will read it. But keep in mind that Google wants to sell us something.

A the time, it was a free language translation tool. You weren't paying for transformers in 2017.

Re: I don't know how you get here from “predict the next word”

#118

Post by qsera »

basch wrote: post #111
qsera wrote: post #98

Earlier quoted context omitted.

>represents UNDERSTANDING of the input language. It does not have an understanding, it pattern matches the "idea shape" of words in the "idea space" of training data and calculates the "idea shape" that is likely to follow considering all the "idea shape" patterns in its training data. It mimics understanding. It feels mysterious to us because we cannot imagine the mapping of a corpus of text to this "idea space". It…

Semantics. Its a encoded position that represents meaning in a way that is useful and reusable. That is "understanding." It's a mathematical representation of grasp.

Yea, semantics is important. It is not "understanding" any more than a microphone+ADC is hearing.

Re: I don't know how you get here from “predict the next word”

#119

Post by 317070 »

sasjaws wrote: post #93

A while ago i did the nanogpt tutorial, i went through some math with pen and paper and noticed the loss function for 'predict the next token' and 'predict the next 2 tokens' (or n tokens) is identical. That was a bit of a shock to me so wanted to share this thought. Basically i think its not unreasonable to say llms are trained to predict the next book instead of single token. Hope this is usefull to someone.

As an expert in the field: this is exactly right.

LLMs are trained to do whole book prediction, at training time we throw in whole books at the time. It's only when sampling we do one or a few tokens at the time.

Re: I don't know how you get here from “predict the next word”

#120

Post by basch »

qsera wrote: post #118
basch wrote: post #111

Earlier quoted context omitted.

Semantics. Its a encoded position that represents meaning in a way that is useful and reusable. That is "understanding." It's a mathematical representation of grasp.

Yea, semantics is important. It is not "understanding" any more than a microphone+ADC is hearing.

agree to disagree. encoding a meaning is understanding. I cited a source using the word in the same way.
Post reply on HN