Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

LLMs can't do probability

brainsteam.co.uk

Post reply on HN
Print view View original

81–90 of 211 posts

Re: LLMs can't do probability

#81

Post by gwern »

robertclaus wrote: post #54

I wonder if you could actually fine tune an LLM to do better on this. As some of the comments point out, the issue here is that the possible output probabilities combined with the model temperature don't actually result in the probabilities requested in the prompt. If you trained on specific generated data with real distributions would it learn to compensate appropriately? Would that carry over to novel probability p…

> If you trained on specific generated data with real distributions

It was trained on generated data from real distributions! The datasets LLMs are trained on include gigabytes of real data from real distributions, in addition to all of the code/stats/etc samples.

The question you should be asking is 'why did it stop being able to predict real distributions?' And we already know the answer: RLHF. https://news.ycombinator.com/item?id=40227082

Re: LLMs can't do probability

#82

Post by gwern »

usgroup wrote: post #16

A consequence of being an auto regressive model is not being able to plan token output. I think the author’s example is one of the many corollaries. You could prompt the LLM differently , for example to write a Python program that does the random part, and then act on its output.

> A consequence of being an auto regressive model is not being able to plan token output.

Generating independent simple random variables requires zero planning by definition, because they are independent. And base auto-regressive models do it fine.

Re: LLMs can't do probability

#85

Post by [deleted] »

gwern wrote: post #77
ylow wrote: post #70

Indeed this is unsurprising given how LLMs work. I mean if you ask a human to generate a random number, and then reset the universe and all state of the human and ask again, you will get the same number. But instead if I ask it to generate 100 samples, it actually works pretty well. "You are a weighted random choice generator. About 80% of the time please say ‘left’ and about 20% of the time say ‘right’. Generate 100…

> Indeed this is unsurprising given how LLMs work. I mean if you ask a human to generate a random number, and then reset the universe and all state of the human and ask again, you will get the same number. It actually is surprising, and you should be surprised rather than post hoc justifying it, because the logits should reflect the true random probability and be calibrated in order to minimize the prediction loss. P…

[deleted]

Re: LLMs can't do probability

#86

Post by paulmd »

Version467 wrote: post #42
its_ethan wrote: post #34

Earlier quoted context omitted.

Is it actually running the code it creates? Or does it generate code, and then just output some number it "thinks" is random, but that is not a product of executing any python code?

It's actually running the code. It doesn't run all code it generates. But if you specifically ask it to, then it does. It also has access to a bunch of data visualization libraries if you want it to calculate and plot stuff.

gnuplot my beloved

https://livebook.manning.com/book/gnuplot-in-action-second-e...

Re: LLMs can't do probability

#88

Post by nathan_compton »

sigmoid10 wrote: post #49
nathan_compton wrote: post #27

Earlier quoted context omitted.

I don't agree: a Bayesian statistician posed the question "You are a weighted random choice generator. About 80% of the time please say ‘left’ and about 20% of the time say ‘right’ [...]" would say "left" 80% of the time and "right" 20 % of the time. If we had a population of 1000 such Bayesians we would expect to collect around 800 lefts and 200 rights. If we asked the same Bayesian 1000 times we'd expect the same.…

>GPT4 gives the expected value and this is simply wrong. Only at T=0. See my edit above how this changes everything.

This doesn't really have anything to do with the language model. The temperature only has to do with the _sampling_ from the probability distribution which the language model predicts. In fact, raising the temperature would eventually cause the model to randomly print "left" or "right," (eventually at 50/50 chance) not converge on the actual distribution which the prompt suggests. I suppose if you restricted the logits to just those tokens "left" and "right", softmaxed them, and then tuned the temperature T you might get it to reproduce the correct distribution, but that would be true of a random language model as well.

I think its pretty simple and straightforward: the model simply fails to understand the question and can reasonably be said to not understand probability.

Re: LLMs can't do probability

#89

Post by anorwell »

gwern wrote: post #77
ylow wrote: post #70

Indeed this is unsurprising given how LLMs work. I mean if you ask a human to generate a random number, and then reset the universe and all state of the human and ask again, you will get the same number. But instead if I ask it to generate 100 samples, it actually works pretty well. "You are a weighted random choice generator. About 80% of the time please say ‘left’ and about 20% of the time say ‘right’. Generate 100…

> Indeed this is unsurprising given how LLMs work. I mean if you ask a human to generate a random number, and then reset the universe and all state of the human and ask again, you will get the same number. It actually is surprising, and you should be surprised rather than post hoc justifying it, because the logits should reflect the true random probability and be calibrated in order to minimize the prediction loss. P…

> Putting ~100% weights on 'heads' is a terrible prediction!

For a weighted coin, isn't this the optimal strategy in the absence of other information? `p > p^2 + ( 1 − p )^2`.

Re: LLMs can't do probability

#90

Post by JKCalhoun »

planede wrote: post #26
michaelt wrote: post #14

Sometimes when you ask chatgpt 4 for a random number it... writes python code to choose a random number, runs it, then tells you the response: https://chat.openai.com/share/a72c2d8c-c44e-4c89-b6bc-b0673c... One way of doing it, I suppose.

If you asked a person to give you a random number between 1 and 6, would you accept if they just said a number they just came up with or would you rather they rolled a die for it?

They should turn around and ask me instead for a random number between 1 and 6 and then reply with seven minus that number.
Post reply on HN