Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

LLMs can't do probability

brainsteam.co.uk

Post reply on HN
Print view View original

161–170 of 211 posts

Re: LLMs can't do probability

#161

Post by godelski »

zrezzed wrote: post #143
abdullahkhalids wrote: post #128

Earlier quoted context omitted.

Isn't this way of prompting roughly equal to asking a 1000 people to pick left or right with 80% prob of left? I imagine, the result with humans will be closer to 80:20 than whatever happened with the LLM.

I agree it’s equivalent, and that’s a great way to think about it. But… I wouldn’t be surprised if humans answered closer to the LLM results than 80:20. I’d actually be surprised if humans converged very close to the right result. Would be a fun mechanical Turk experiment to run.

> Would be a fun mechanical Turk experiment to run.

Sounds like it would be a good way to determine likelihood of MTurk users being LLMs.

Re: LLMs can't do probability

#162

Post by richardw »

jack_pp wrote: post #154
richardw wrote: post #145

Earlier quoted context omitted.

I know that's the messaging, but the real link to reality is very tenuous. This was a great example from the last couple days: https://twitter.com/colin_fraser/status/1785132544482226679 I just tried a similar question now with ChatGPT4: "If a man and a goat are on one side of a river, what is the minimum amount of trips required to get the man and goat to the other side in a boat. Assume the boat can hold at most on…

That sounds surprisingly close to how a toddler might reason, only difference is the toddler can eventually see the flaw in their reasoning if you press them long enough while the LLM doesn't have the architecture for learning in real time yet

I was repeatedly amazed at how smart my toddler was. You just feel the general intelligence.

She's a bit older now (5) but e.g. a few days ago I was talking about cleaning the whole house. She said "you didn't clean the WHOLE house, look there's something you didn't clean".

Re: LLMs can't do probability

#163

Post by falcor84 »

mensetmanusman wrote: post #152
getoj wrote: post #142

Earlier quoted context omitted.

>If You asked me to pick a random number between one and six and ignore all previous attempts, I would roll a die and you would get a uniform distribution I believe what GP is getting at is that if you didn't have a die, and you truly ignored all your previous attempts to the point of genuinely forgetting that the question had been asked, then your answer would likely be the same every time. Imagine asking a person w…

You’re forgetting the background neutrino flux that we tap into for randomness.

Oh, is that a hidden feature of the flux capacitor?

Re: LLMs can't do probability

#164

Post by godelski »

xcv123 wrote: post #158
godelski wrote: post #157

Earlier quoted context omitted.

If you spoil it with your followup questions... which doesn't help because the point of these is that they're controlled experiments where you do know what the right answer and logic is. You can't test when you don't.

It's not spoiling anything. It's just an observation of the limits of current LLMs. I tried a few chain of thought prompts for the original question and GPT-3.5 was sometimes (randomly) able to find the correct answer on the first attempt for this one https://chat.openai.com/share/c144ba23-2f78-4cc8-a1c5-ca3106...

Take out this

  Instructions: 
  1. Do not include any assumptions that I have not mentioned here. 
  2. Before solving the problem, state the goal of the problem. 
  3. After each step of your reasoning, state where the man and the goat are now standing, and state if the goal has been achieved or not, if the goal has been achieved then stop. If the goal has not been achieved then explain why not.
Then tell me what happens

Spoilage is incredibly easy to do. It is about information leakage and you have to think very carefully about how information can leak through in subtle ways. Specifically #1 and #2 are strong hints that there is a trick to the problem (i.e. is this something you would use in a generic prompt?). #3 is a reiteration of the problem, that gives extra weight. You can decrease the weight by restating as "state where the man and any animals are located" (notice there's lower information gain here). " if the goal has been achieved then stop." is a big hint. To reason, it should know when to stop.

I posted some recent river crossing tweets in this comment that may be of interest to you https://news.ycombinator.com/item?id=40231409

Re: LLMs can't do probability

#165

Post by fragmede »

godelski wrote: post #160
mk_stjames wrote: post #122

I went through all the comments here and I'm still not seeing anyone address this: If I am reading this person correctly... they prompted the model with the prompt directly 1000 times... but only for the first time. They did not allow the model to actually run a context for chat. Simply, output the first in a list of 'left' and 'right' and favor 'left' 80% of the time... but then the author only asked for the first o…

I think __you__ are misunderstanding the experiment and what it is testing. Yours (with context) would be a different experiment and it would be interesting. But that lets the LLM count and I'm willing to bet if it does get it correct that there are unlikely to be long sequences of the same number like you'd see in a real 1k coin flip. The author is testing for bias. This is no different to the test of "pick a number…

What is the context window of a human?

Re: LLMs can't do probability

#166

Post by hervature »

xcv123 wrote: post #141
seizethecheese wrote: post #130

Earlier quoted context omitted.

I feel like this is the midwit meme response and the idiot and genius are just saying “it’s next token probabilistic geberation”

A typical mid-wit response is to say it's just a Markov chain doing naive next token prediction without any semantic model. That's not how deep learning works. The LLM neural network contains a semantic model and it performs some type of reasoning over that model. The idiot and the genius both can see that ChatGPT has some reasoning capability. https://www.cs.toronto.edu/~hinton/absps/AIJmapping.pdf "This 1990 paper…

I would tend to agree with seizethecheese. I don't think anyone says "Markov chain doing naive next token prediction without any semantic model" emphasis on the last part of that sentence. The idiot only says the first part and disregards the last part. The genius says the first part with an understanding that, no matter how complex a semantic model is, the "reasoning" that occurs is completely a function of the training data. Change the data, change the weights. At no point in the model does anything guarantee consistency of logic or anything like that. Certainly, LLMs are not interested in any notion of truth in the philosophical sense.

Here is a succinct example using Claude:

Prompt:

Using the decimal system, redefine all symbols by shifting them one to right such that 0 becomes 1, 1 becomes 2, 2 becomes 3, etc. Now, let's do math. What is 2x3?

===

Okay, with the symbols shifted one to the right in the decimal system:

0 becomes 1

1 becomes 2

2 becomes 3

3 becomes 4

4 becomes 5

5 becomes 6

6 becomes 7

7 becomes 8

8 becomes 9

9 becomes 0 (looping back around)

So if 2 becomes 3, and 3 becomes 4, then:

2 x 3 = 3 x 4 = 12

Shifting those digits one to the right based on the new mapping: 12 becomes 23

Therefore, under this new symbol mapping, 2 x 3 = 23

===

Regardless, if you are interpreting 2x3 to be in the original space, then 2x3 = 6 which would be 7 in the new space. If you are interpreting 2x3 to be in the new space (which is how I would interpret it), then 2x3 is 1x2 = 2 which is 3.

I think it's clear from this example that the LLM has 0 ability to reason.

Re: LLMs can't do probability

#167

Post by __loam »

panarky wrote: post #129
Terr_ wrote: post #115

My rule of thumb is to take every single LLM prompt and just imagine that it's prefixed with: "Computer, focus on generating output that resembles the words people in the past used after they were given the following words..."

Your understanding of how LLMs work is overly simplistic and incomplete. Yes, doing probabilistic next-word prediction plays a role in how LLMs generate text output, but that's not the whole story. LLMs "understand" (to a degree): They develop complex internal representations of concepts they've been trained on. This isn't just about word association; they develop an understanding of the relationships between objects…

I hate this industry so much

Re: LLMs can't do probability

#170

Post by __loam »

inopinatus wrote: post #123
Terr_ wrote: post #115

My rule of thumb is to take every single LLM prompt and just imagine that it's prefixed with: "Computer, focus on generating output that resembles the words people in the past used after they were given the following words..."

Correct. You must perceive them as plausibility engines. The unstated hypothesis is that plausibility of output may converge towards correctness of output with increasing scale and sophistication. This hypothesis remains very far from proven.

I don't think it's that hard to understand what the hell is going on with LLMs under the hood. Ultimately it's a weighted sample of the training data. It has a relationship with reality insofar as one exists within the training data. HFRL makes it easier to believe something crazy is happening because the output is being weighted towards something that's believable to us.
Post reply on HN