Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

LLMs can't do probability

brainsteam.co.uk

Post reply on HN
Print view View original

201–210 of 211 posts

Re: LLMs can't do probability

#201

Post by imtringued »

xcv123 wrote: post #183
krainboltgreene wrote: post #180

Earlier quoted context omitted.

Anyone who has studied neural networks also knows there's no comparison between computer neural networks and human biological neural networks. The name was picked because of a passing familiarity with the biological by someone who didn't have any experience in biological neural networks. It's been sufficiently proven they have no similarity by countless academics.

That is a blatant oversimplification and not true. There are both similarities and differences. New ANN training methods are inspired by studies of biological neural networks (Dropout Regularization is one example)

You can't implement backpropagation biologically. The fact that you didn't even mention spiking neural networks speaks volumes. Those are heavily biologically inspired and yet they have fallen behind ANNs precisely because backpropagation doesn't work on them.

Re: LLMs can't do probability

#203

Post by gwern »

anorwell wrote: post #89
gwern wrote: post #77

Earlier quoted context omitted.

> Indeed this is unsurprising given how LLMs work. I mean if you ask a human to generate a random number, and then reset the universe and all state of the human and ask again, you will get the same number. It actually is surprising, and you should be surprised rather than post hoc justifying it, because the logits should reflect the true random probability and be calibrated in order to minimize the prediction loss. P…

> Putting ~100% weights on 'heads' is a terrible prediction! For a weighted coin, isn't this the optimal strategy in the absence of other information? `p > p^2 + ( 1 − p )^2`.

No, because you're confusing loss functions: a LLM makes a probabilistic prediction, not a hard decision. That is the optimal strategy only if you have something like a 0-1 loss function†, akin to betting on a coin flip, which is not a proper scoring rule (and not easily differentiable either).

Whereas LLMs are usually trained with a proper scoring rule which incentivizes them to report calibrated predictions, like mean squared error. For that, the optimal prediction is just '50%', perhaps transformed into log-odds, and whatever the equivalent of '50%' is over the BPE vocabulary.

† eg if you are betting $1 on whether heads or tails come up, it is true that you can't do better than always betting $1 on the side with P>50% - and strikingly, this is not what people do in setups like the spinner game (or Twitter polls), they 'probability match', which is optimal in terms of Thompson sampling, as if they were playing a indefinitely-long repeated bandit to minimize regret. I usually take this as an example of System I vs System II: showing how hard it is to break our real-world-appropriate intuitive behavior in artificial game setups. If you think about it, in the usual spinner-game, probability matching is just straightforwardly wrong and it's not like a bandit at all; but you do have to think about it.

Re: LLMs can't do probability

#204

Post by gwern »

buescher wrote: post #149
gwern wrote: post #81

Earlier quoted context omitted.

> If you trained on specific generated data with real distributions It was trained on generated data from real distributions! The datasets LLMs are trained on include gigabytes of real data from real distributions, in addition to all of the code/stats/etc samples. The question you should be asking is 'why did it stop being able to predict real distributions?' And we already know the answer: RLHF. https://news.ycombin…

Do we know in any detail who provided the RLHF and according to what rules for any of these models?

No, not really. OA has been reticent to publish any real details about what RLHF GPT-4 and later models go through; while some models have been much more open, those weren't used in OP.

And it's unclear how easily you can interrogate their code/data to understand exactly how the RLHF goes wrong here - it seems unlikely that there are all that many raters rewarding conversations with heads rather than tails in hypothetical coinflips, so it's probably a more subtle issue of entropy collapse. (It's not that easy to understand why DL stuff does the stuff it does, and it's even more true that when it comes to RL stuff, it's much easier to observe outcomes than to understand how exactly the RL process yielded that outcome.)

So, we can see the effects before/after very clear in the OA Figure 8 graph in https://arxiv.org/pdf/2303.08774.pdf#page=12&org=openai on calibration, but I dunno if even they could tell you what exactly about the raters or PPO hyperparameters or whatever causes that.

Re: LLMs can't do probability

#205

Post by NoPicklez »

mch82 wrote: post #102
offmycloud wrote: post #92

LLMs can't do math in general, they need external help to do simple math problems with any consistency.

Can you recommend any references that explain why LLMs can’t do math?

My understanding was that LLM's don't actually know or understand math, physics etc but language.

It's only with the introduction of things like Wolfram Alpha into ChatGPT for example that they can actually perform math with accuracy, because it's being passed off.

Re: LLMs can't do probability

#206

Post by __loam »

dartos wrote: post #199
dTal wrote: post #174

Earlier quoted context omitted.

Depending on what you mean by "weighted sample", that's either trivially true (the network is of course a function of its training data and nothing else) or trivially false (the network generalizes over the training data and has not memorized it). It is not a good intuition pump for why an LLM can hold up one end of a conversation, or follow novel instructions - it is not reading from a script, nor regurgitating chop…

> this is true of anything that learns Sure, but most things that learn have actual reality as a training set. LLMs have human curated data, which isn’t and can’t be perfectly representative of reality.

Couldn't have said it better myself.

Re: LLMs can't do probability

#208

Post by DougBTX »

nextaccountic wrote: post #67
DougBTX wrote: post #47

Earlier quoted context omitted.

Yes, probably. At temperature zero the model will be completely deterministic, so a particular prompt will always produce the same result (ignoring for a second that some fairly common optimisations introduce data races in the GPU). On the other hand, does it really matter? With a slight tweak to the prompt, ChatGPT generates some serviceable code: > Run a function to produce a random number between 1 and 10. What is…

> (ignoring for a second that some fairly common optimisations introduce data races in the GPU). Okay so are any GPU compilers intentionally introducing data races in programs that previously exhibited no data races?

Not really compilers, but the underlying GPU libraries.

Here’s a good jumping off point: https://pytorch.org/docs/stable/generated/torch.use_determin...

Re: LLMs can't do probability

#209

Post by JKCalhoun »

awesomeMilou wrote: post #197
JKCalhoun wrote: post #90

Earlier quoted context omitted.

They should turn around and ask me instead for a random number between 1 and 6 and then reply with seven minus that number.

How is that more random?

I was being somewhat tongue in cheek (absurdist?), but I suppose if you don't know my algorithm (7 - n) then it does add layer of ... uncertainty?

Re: LLMs can't do probability

#210

Post by xcv123 »

imtringued wrote: post #201
xcv123 wrote: post #183

Earlier quoted context omitted.

That is a blatant oversimplification and not true. There are both similarities and differences. New ANN training methods are inspired by studies of biological neural networks (Dropout Regularization is one example)

You can't implement backpropagation biologically. The fact that you didn't even mention spiking neural networks speaks volumes. Those are heavily biologically inspired and yet they have fallen behind ANNs precisely because backpropagation doesn't work on them.

That is missing the forest for the trees. Human cognition is more similar to a learned connectionist model versus an explicitly programmed rules-based model.
Post reply on HN