[deleted]
I don't know how you get here from “predict the next word”
151–160 of 275 posts
Re: I don't know how you get here from “predict the next word”
#152Earlier quoted context omitted.
As an expert in the field: this is exactly right. LLMs are trained to do whole book prediction, at training time we throw in whole books at the time. It's only when sampling we do one or a few tokens at the time.
where do you get these books? honking intensifies WHERE DO YOU GET THESE BOOKS?!
The local library.
Re: I don't know how you get here from “predict the next word”
#153Earlier quoted context omitted.
As an expert in the field: this is exactly right. LLMs are trained to do whole book prediction, at training time we throw in whole books at the time. It's only when sampling we do one or a few tokens at the time.
where do you get these books? honking intensifies WHERE DO YOU GET THESE BOOKS?!
Can anyone even say what a book really is at the end of the day? It's such an abstract concept. /s
Re: I don't know how you get here from “predict the next word”
#154Is this an ad? Seems like it. The text is not really what the headline suggests.
Do you think the submitter intended this as an ad? His post history doesn't seem suspicious. Or do you think article's author wrote this an an ad? He's a reputable academic who seems impressed with an AI tool he used and is honestly sharing his thoughts. For reference he published the 80 page inflation mini-book 2 weeks ago asking for feedback: https://www.grumpy-economist.com/p/inflation
> Or do you think article's author wrote this an an ad? He's a reputable academic who seems impressed with an AI tool he used and is honestly sharing his thoughts.
Ghuntley used to be reputable on here, then the crypto money looked too juicy.
Re: I don't know how you get here from “predict the next word”
#155Earlier quoted context omitted.
>represents UNDERSTANDING of the input language. It does not have an understanding, it pattern matches the "idea shape" of words in the "idea space" of training data and calculates the "idea shape" that is likely to follow considering all the "idea shape" patterns in its training data. It mimics understanding. It feels mysterious to us because we cannot imagine the mapping of a corpus of text to this "idea space". It…
I am not knowledgeable on how transformer works but, what if, us humans just do the same thing in our minds as well ? What if our feeling of "understanding" is merely just the emotional response to a pattern matching as you just said?
Yea, you said it. It is the feeling of understanding and feeling/sensing implies consciousness. Why does it matter? I don't know. All I know is that it is not the same thing, because a chunk of metal cannot feel. So I don't want it to be called by the same name.
When AI marketing (ab)uses the word, it is to project the appearance of human equivalence. And I don't like to fall for it.
Re: I don't know how you get here from “predict the next word”
#156[flagged]
Re: I don't know how you get here from “predict the next word”
#157I think it’s funny that at Google I invented and productized next word (and next action) predictor in Gmail and hangouts chat and I’ve never had a single person come to me and ask how this all works. To me LLMs are incredibly simple. Next word next sentence next paragraph and next answer are stacked attention layers which identify manifolds and run in reverse to then keep the attention head on track for next token. I…
A lot of people in tech thrive on the mystery and don't like explaining things in simple terms. It makes what they do seem more valuable if no one can understand what they're talking about. At the same time, being vague and mysterious can help hide someone's own misunderstandings. When you speak clearly you need to be accurate, because it's more obvious when you're wrong.
Re: I don't know how you get here from “predict the next word”
#158Earlier quoted context omitted.
>agree to disagree. Yea >encoding a meaning is understanding. encoding a meaning is encoding. Nothing more!
what is understanding but encoded meaning distilled into pure structure, in both cases a property of a pattern? No need to gatekeep the word "understanding" behind subjective human experience eg qualia.
> No need to gatekeep the word "understanding" behind subjective human experience eg qualia.
Yea, I think gatekeeping is needed exactly for the same reason. Make up another word if you want..
Re: I don't know how you get here from “predict the next word”
#159Earlier quoted context omitted.
As an expert in the field: this is exactly right. LLMs are trained to do whole book prediction, at training time we throw in whole books at the time. It's only when sampling we do one or a few tokens at the time.
where do you get these books? honking intensifies WHERE DO YOU GET THESE BOOKS?!
[deleted]
Re: I don't know how you get here from “predict the next word”
#160Earlier quoted context omitted.
As an expert in the field: this is exactly right. LLMs are trained to do whole book prediction, at training time we throw in whole books at the time. It's only when sampling we do one or a few tokens at the time.
where do you get these books? honking intensifies WHERE DO YOU GET THESE BOOKS?!
We do things, but it doesn't feel right