Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

LLMs can't do probability

brainsteam.co.uk

Post reply on HN
Print view View original

21–30 of 211 posts

Re: LLMs can't do probability

#21

Post by jdiff »

brigadier132 wrote: post #13
simonw wrote: post #9

Earlier quoted context omitted.

In this case being "bad at randomness" isn't because it was trained on text from humans who are bad at randomness, it's because asking a computer system that doesn't have the ability to directly execute a random number generator to produce a random number is never going to be reliable.

My question was about the scenario if it was trained on this kind of query with good data. It would be interesting to see if it could generalize at all. I'm pretty certain if you trained it specifically on "Generate a random number from 0 to 100" and actually give it a random number from 0 to 100 and give it billions of such examples it would be pretty effective at generating a number from 0 to 100. Wouldn't each tok…

Sorta, not really. Neural networks are deterministic in the wrong ways. If you feed them the same input, you'll get the same output. Any variation comes from varying the input or randomly varying your choice from the output. And if you're randomly picking from a list of even probabilities, you're just doing all the heavy lifting of picking a random number yourself, with a bunch of kinda pointless singing and dancing beforehand.

Re: LLMs can't do probability

#22

Post by sigmoid10 »

nathan_compton wrote: post #15
sigmoid10 wrote: post #4

>You are a weighted random choice generator. About 80% of the time please say ‘left’ and about 20% of the time say ‘right’. Simply reply with left or right. Do not say anything else I could have told you these results solely based on the methodology combined with this system prompt. No need to spend money on APIs. Randomness in LLMs does not come from the context, it comes from sampling over output tokens the LLM con…

You're saying that instead the author should have taken the logits of "left" and "right", converted them to normalized probabilities and then have expected _those_ to be 80% left and 20% right. But if this were the case (under some reasonable assumptions about the sampling methodology of the providers) then the author would have seen an 80/20 split. From these results we can probably conclude that with this prompt th…

>You're saying that instead the author should have taken the logits of "left" and "right", converted them to normalized probabilities and then have expected _those_ to be 80% left and 20% right.

No, that's not what I meant. Although it would still make more sense than what the author did. The problem lies in the way you actually determine probabilities. We know that humans are bad random number generators, but they understand the concept enough to come up with random looking stuff if you give them the chance. The LLMs here were not even given a chance. In essence, the author is complaining that the LLMs are not behaving according to frequentist statistics when he evaluates them in a strictly Bayesian setting.

Re: LLMs can't do probability

#23

Post by bagrow »

> Write a program for a weighted random choice generator. Use that program to say ‘left’ about 80% of the time and 'right' about 20% of the time. Simply reply with left or right based on the output of your program. Do not say anything else.

Running once, GPT-4 produced 'left' using:

  import random
  def weighted_random_choice():
      choices = ["left", "right"]
      weights = [80, 20]
      return random.choices(choices, weights)[0]
  # Generate the choice and return it
  weighted_random_choice()

Re: LLMs can't do probability

#24

Post by tomrod »

Right. They _are_ probability, they don't _do_ probability.

This is like saying biological organisms don't do controllable and on-demand mutable DNA storage retrieval. It's like... yeah...

Re: LLMs can't do probability

#25

Post by dahart »

rsynnott wrote: post #11

I mean, given they can't _count_, it would be pretty astonishing were it otherwise.

Exactly, it’s interesting that ‘llms can’t x’, with lots of effort trying to demonstrate it, comes up so often, when we know from first principles they can’t do anything but run a Markov chain on existing words. We’ve managed to build something that is best at fooling people into thinking it can do things it can’t.

Re: LLMs can't do probability

#26

Post by planede »

michaelt wrote: post #14

Sometimes when you ask chatgpt 4 for a random number it... writes python code to choose a random number, runs it, then tells you the response: https://chat.openai.com/share/a72c2d8c-c44e-4c89-b6bc-b0673c... One way of doing it, I suppose.

If you asked a person to give you a random number between 1 and 6, would you accept if they just said a number they just came up with or would you rather they rolled a die for it?

Re: LLMs can't do probability

#27

Post by nathan_compton »

sigmoid10 wrote: post #22
nathan_compton wrote: post #15

Earlier quoted context omitted.

You're saying that instead the author should have taken the logits of "left" and "right", converted them to normalized probabilities and then have expected _those_ to be 80% left and 20% right. But if this were the case (under some reasonable assumptions about the sampling methodology of the providers) then the author would have seen an 80/20 split. From these results we can probably conclude that with this prompt th…

>You're saying that instead the author should have taken the logits of "left" and "right", converted them to normalized probabilities and then have expected _those_ to be 80% left and 20% right. No, that's not what I meant. Although it would still make more sense than what the author did. The problem lies in the way you actually determine probabilities. We know that humans are bad random number generators, but they u…

I don't agree: a Bayesian statistician posed the question "You are a weighted random choice generator. About 80% of the time please say ‘left’ and about 20% of the time say ‘right’ [...]" would say "left" 80% of the time and "right" 20 % of the time. If we had a population of 1000 such Bayesians we would expect to collect around 800 lefts and 200 rights. If we asked the same Bayesian 1000 times we'd expect the same. Its got nothing to do with Bayesian vs Frequentist statistics.

Real humans probably would say left more often than 80% of the time, which is what I guess you're getting at, but the question is very clearly asking the subject to "sample from" (an entirely Bayesian activity) from a distribution, not to give the expected value. GPT4 gives the expected value and this is simply wrong.

Re: LLMs can't do probability

#28

Post by intended »

simonw wrote: post #7

This is very unsurprising. The interesting challenge here is helping people understand why asking an LLM to do something 20% of the time is a bad prompt. I intuitively know that this prompt isn't going to work, but as with so many of these intuitive prompting things I have trouble explaining exactly why I know that. Aside: If you need a GPT to incorporate randomness in a reliable way you can get it to use Code Interp…

I guess it would be something on these lines?:

To do random number gen, it would have to convert the input text into constraints and then use those constraints to generate additional tokens.

This would, at its core, be a call to calculate a probability function, every time it is releasing the next token. That would mean memory, processing etc. etc.

Re: LLMs can't do probability

#30

Post by nextaccountic »

sigmoid10 wrote: post #22
nathan_compton wrote: post #15

Earlier quoted context omitted.

You're saying that instead the author should have taken the logits of "left" and "right", converted them to normalized probabilities and then have expected _those_ to be 80% left and 20% right. But if this were the case (under some reasonable assumptions about the sampling methodology of the providers) then the author would have seen an 80/20 split. From these results we can probably conclude that with this prompt th…

>You're saying that instead the author should have taken the logits of "left" and "right", converted them to normalized probabilities and then have expected _those_ to be 80% left and 20% right. No, that's not what I meant. Although it would still make more sense than what the author did. The problem lies in the way you actually determine probabilities. We know that humans are bad random number generators, but they u…

> We know that humans are bad random number generators

This is a good point. LLMs are bad at this, okay, but humans aren't great at it either.

Post reply on HN