Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

I don't know how you get here from “predict the next word”

grumpy-economist.com

Post reply on HN
Print view View original

141–150 of 275 posts

Re: I don't know how you get here from “predict the next word”

#141

Post by echelon »

qsera wrote: post #132
echelon wrote: post #110

Earlier quoted context omitted.

The real money is in using the models to build utility and money-making companies. You're removed from orders of magnitude in upside potential if you have to wait for the public markets.

> money-making companies You mean, money sucking companies, right? >You're removed from orders of magnitude in upside potential if you have to wait for the public markets. because that won't work. That is why!

> You mean, money sucking companies, right?

Is that what you (and all people) are in your job function? A money suck?

Do you ever buy anything for food, shelter, and clothing? Do you have hobbies?

Capitalism means we don't have to all be hunter-gatherers, and I'm pretty keen on that trade.

> because that won't work. That is why!

This is the forum for a venture capital firm. A lot of the folks here build things with the intention of creating value and getting compensated for that value creation. Other valid options are sitting at home and playing video games, reading a book, or posting on HN.

I like working on problems where I'm the customer and where I would buy the product if it existed. Turns out, there tend to be other people who would buy my software too.

Re: I don't know how you get here from “predict the next word”

#142

Post by nairboon »

c22 wrote: post #76
wavemode wrote: post #74

Earlier quoted context omitted.

When I say "model" here I'm referring to any statistical model (in this example, probably linear regression). Not specifically large language models / neural networks.

Gotcha, I don't think I know enough about it. What constitutes training data for a for a (non neural network) statistical model? Is this something I could play around with myself with pen and paper?

Just the raw numbers? You list the y's and the x's and the model is approximating y=f(x) from the above example. You can totally do it with pen and paper. This is what it'd look like (for linear regression): https://observablehq.com/@yizhe-ang/interactive-visualizatio...

Re: I don't know how you get here from “predict the next word”

#143

Post by basch »

qsera wrote: post #128
basch wrote: post #120

Earlier quoted context omitted.

agree to disagree. encoding a meaning is understanding. I cited a source using the word in the same way.

>agree to disagree. Yea >encoding a meaning is understanding. encoding a meaning is encoding. Nothing more!

what is understanding but encoded meaning distilled into pure structure, in both cases a property of a pattern?

No need to gatekeep the word "understanding" behind subjective human experience eg qualia.

Re: I don't know how you get here from “predict the next word”

#144

Post by Kim_Bruning »

wavemode wrote: post #24

> the kind of analysis the program is able to do is past the point where technology looks like magic. I don’t know how you get here from “predict the next word.” You're implicitly assuming that what you asked the LLM to do is unrepresented in the training data. That assumption is usually faulty - very few of the ideas and concepts we come up with in our everyday lives are truly new. All that being said, the refine.in…

> You're implicitly assuming that what you asked the LLM to do is unrepresented in the training data. That assumption is usually faulty - very few of the ideas and concepts we come up with in our everyday lives are truly new.

I made a cursed CPU in the game 'Turing Complete'; and had an older version of claude build me an assembler for it?

Good luck finding THAT in the training data. :-P

(just to be sure, I then had it write actual programs in that new assembly language)

Re: I don't know how you get here from “predict the next word”

#145

Post by TuringTest »

317070 wrote: post #119
sasjaws wrote: post #93

A while ago i did the nanogpt tutorial, i went through some math with pen and paper and noticed the loss function for 'predict the next token' and 'predict the next 2 tokens' (or n tokens) is identical. That was a bit of a shock to me so wanted to share this thought. Basically i think its not unreasonable to say llms are trained to predict the next book instead of single token. Hope this is usefull to someone.

As an expert in the field: this is exactly right. LLMs are trained to do whole book prediction, at training time we throw in whole books at the time. It's only when sampling we do one or a few tokens at the time.

Isn't that the same as compressing the whole book, in a special differential format that compares how the text looks from any given point before and after?

Re: I don't know how you get here from “predict the next word”

#146

Post by Morromist »

visarga wrote: post #33

> Nothing you write will matter if it is not quickly adopted to the training dataset. That is my take too, I was surprised to see how many people object to their works being trained on. It's how you can leave your mark, opening access for AI, and in the last 25 years opening to people (no restrictions on access, being indexed in Google).

"On reflection I have started to worry again. In 10 to 20 years nobody will read anything any more, they just will read LLM digests. So, the single most important task of a writer starting right now is to get your efforts wired in to the LLMs"

You're words will be like a drop in the ocean, an ocean where the water volume keeps increasing every year. Also if nobody reads anything anymore what's the point?

Re: I don't know how you get here from “predict the next word”

#147

Post by arkh »

wavemode wrote: post #41
selridge wrote: post #34

Earlier quoted context omitted.

>You're implicitly assuming that what you asked the LLM to do is unrepresented in the training data. This is just as stuck in a moment in time as "they only do next word prediction" What does this even mean anymore? Are we supposed to believe that a review of this paper that wasn't written when that model (It's putatively not an "LLM", but IDK enough about it to be pushy there) was trained? Does that even make sense?…

Statistical models generalize. If you train a model that f(x) = 5 and f(x+1) = 6, the number 7 doesn't have to exist in the training data for the model to give you a correct answer for f(x+2) Similarly, if there are millions of academic papers and thousands of peer reviews in the training data, a review of this exact paper doesn't need to be in there for the LLM to write something convincing. (I say "convincing" rath…

I expected (and still expect) a lot from LLM with cross disciplinary research.

I think they should be the perfect tool to find methods or results in a field which look like it could be used in another field.

Re: I don't know how you get here from “predict the next word”

#148

Post by [deleted] »

Kim_Bruning wrote: post #144
wavemode wrote: post #24

> the kind of analysis the program is able to do is past the point where technology looks like magic. I don’t know how you get here from “predict the next word.” You're implicitly assuming that what you asked the LLM to do is unrepresented in the training data. That assumption is usually faulty - very few of the ideas and concepts we come up with in our everyday lives are truly new. All that being said, the refine.in…

> You're implicitly assuming that what you asked the LLM to do is unrepresented in the training data. That assumption is usually faulty - very few of the ideas and concepts we come up with in our everyday lives are truly new. I made a cursed CPU in the game 'Turing Complete'; and had an older version of claude build me an assembler for it? Good luck finding THAT in the training data. :-P (just to be sure, I then had…

[deleted]

Re: I don't know how you get here from “predict the next word”

#149

Post by vasco »

qsera wrote: post #92
vasco wrote: post #59

Earlier quoted context omitted.

My neurons are also just passing electric signals back and forward and exchanging water and salts with the rest of my body.

> just passing electric signals back and forward Ok, feel free to call yourselves a toaster, I don't mind!

What, reductionism only works when you do it?

Re: I don't know how you get here from “predict the next word”

#150

Post by orf »

compass_copium wrote: post #79
kristiandupont wrote: post #73

Earlier quoted context omitted.

I had Claude help me get a program written for Linux to compile on macOS. The program is written in a programming language the author invented for the project, a pretty unusual one (for example, it allows spaces in variable names). Claude figured out how the language worked and debugged segfaults until the compiler compiled, and then until the program did. That might not be magic , but it shows a level of sophisticat…

But the programming language has explicitly laid out rules. It was not trained on those sets of rules, but it was trained on many trillions of lines of code. It has a map of how programs work, and an explanation of this new language. It's using training data and data it's fed to generate that result.

That’s still over-general to the point of being useless.

What you wrote would apply to a human approaching this task as well, sans the “many trillion lines of code”.

Post reply on HN