Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

LLMs can't do probability

brainsteam.co.uk

Post reply on HN
Print view View original

171–180 of 211 posts

Re: LLMs can't do probability

#171

Post by __loam »

causal wrote: post #133
panarky wrote: post #129

Earlier quoted context omitted.

Your understanding of how LLMs work is overly simplistic and incomplete. Yes, doing probabilistic next-word prediction plays a role in how LLMs generate text output, but that's not the whole story. LLMs "understand" (to a degree): They develop complex internal representations of concepts they've been trained on. This isn't just about word association; they develop an understanding of the relationships between objects…

Yeah I think the word probabilistic has been broadly misconstrued, because we usually see it in contexts where probabilities are used as shortcuts to avoid comprehensive analysis. But a probability can also be the output of a detailed analysis with rich semantic representation of causes.

I mean, that's exactly what LLMs are. In the absence of any real understanding of cognition, we are just throwing shit at the wall and seeing what the probabilistic model does with the massive amount of data we give it. A definitive analysis, were one even possible with our current models of computing, would probably outperform an LLM.

Re: LLMs can't do probability

#172

Post by deepsquirrelnet »

This is uninteresting. The token logits from the model output are deterministic, while the sampler can be made to simulate probabilistic responses by adding randomization to the token selection (using a deterministic probability distribution). For repeated measurements, unless you have a high amount of randomization in your sampler, you should not expect different answers.

Generate a random seed and write it into your prompt (new seed each prompt). I bet dollars to donuts you’ll get different results.

Re: LLMs can't do probability

#173

Post by xcv123 »

godelski wrote: post #164
xcv123 wrote: post #158

Earlier quoted context omitted.

It's not spoiling anything. It's just an observation of the limits of current LLMs. I tried a few chain of thought prompts for the original question and GPT-3.5 was sometimes (randomly) able to find the correct answer on the first attempt for this one https://chat.openai.com/share/c144ba23-2f78-4cc8-a1c5-ca3106...

Take out this Instructions: 1. Do not include any assumptions that I have not mentioned here. 2. Before solving the problem, state the goal of the problem. 3. After each step of your reasoning, state where the man and the goat are now standing, and state if the goal has been achieved or not, if the goal has been achieved then stop. If the goal has not been achieved then explain why not. Then tell me what happens Spoi…

Yes we know the current LLMs cannot solve the original prompt. That's why I experimented with different prompts.

The instructions are prompting it to proceed rigorously, as it is a logical problem, not a natural language problem. These models are primarily trained for solving natural language processing tasks, and so they are predisposed to answer in a certain way through training and tuning. The models produce less verbose output by default to reduce cost (each token costs money). Telling the model to generate more tokens in step-by-step reasoning enables it to "think" further as it can only "think" when generating each token.

OpenAI could train or tune ChatGPT to "spoil" itself by default when answering any problem that it identifies as a logic problem. It is somewhat arbitrary.

Re: LLMs can't do probability

#174

Post by dTal »

__loam wrote: post #170
inopinatus wrote: post #123

Earlier quoted context omitted.

Correct. You must perceive them as plausibility engines. The unstated hypothesis is that plausibility of output may converge towards correctness of output with increasing scale and sophistication. This hypothesis remains very far from proven.

I don't think it's that hard to understand what the hell is going on with LLMs under the hood. Ultimately it's a weighted sample of the training data. It has a relationship with reality insofar as one exists within the training data. HFRL makes it easier to believe something crazy is happening because the output is being weighted towards something that's believable to us.

Depending on what you mean by "weighted sample", that's either trivially true (the network is of course a function of its training data and nothing else) or trivially false (the network generalizes over the training data and has not memorized it). It is not a good intuition pump for why an LLM can hold up one end of a conversation, or follow novel instructions - it is not reading from a script, nor regurgitating chopped up pieces of text like a Markov chain. It is doing something very clever in a way that is not obvious.

>It has a relationship with reality insofar as one exists within the training data

This is true of anything that learns.

Re: LLMs can't do probability

#175

Post by xcv123 »

hervature wrote: post #166
xcv123 wrote: post #141

Earlier quoted context omitted.

A typical mid-wit response is to say it's just a Markov chain doing naive next token prediction without any semantic model. That's not how deep learning works. The LLM neural network contains a semantic model and it performs some type of reasoning over that model. The idiot and the genius both can see that ChatGPT has some reasoning capability. https://www.cs.toronto.edu/~hinton/absps/AIJmapping.pdf "This 1990 paper…

I would tend to agree with seizethecheese. I don't think anyone says "Markov chain doing naive next token prediction without any semantic model " emphasis on the last part of that sentence. The idiot only says the first part and disregards the last part. The genius says the first part with an understanding that, no matter how complex a semantic model is, the "reasoning" that occurs is completely a function of the tra…

> I think it's clear from this example that the LLM has 0 ability to reason.

It's not a 0 or 1. You are oversimplifying it. Obviously neural networks can learn to generalize patterns of reasoning inferred from their training data. We know that they are not using explicitly defined formal systems of reasoning, and they have some limitations compared to those systems. Anyone who seriously studied neural networks or machine learning understands this.

By the same logic, practically every human on Earth has "0 ability to reason" as their biological neural network will get confused and make mistakes.

Re: LLMs can't do probability

#176

Post by godelski »

fragmede wrote: post #165
godelski wrote: post #160

Earlier quoted context omitted.

I think __you__ are misunderstanding the experiment and what it is testing. Yours (with context) would be a different experiment and it would be interesting. But that lets the LLM count and I'm willing to bet if it does get it correct that there are unlikely to be long sequences of the same number like you'd see in a real 1k coin flip. The author is testing for bias. This is no different to the test of "pick a number…

What is the context window of a human?

Humans don't work like that

Re: LLMs can't do probability

#177

Post by godelski »

xcv123 wrote: post #173
godelski wrote: post #164

Earlier quoted context omitted.

Take out this Instructions: 1. Do not include any assumptions that I have not mentioned here. 2. Before solving the problem, state the goal of the problem. 3. After each step of your reasoning, state where the man and the goat are now standing, and state if the goal has been achieved or not, if the goal has been achieved then stop. If the goal has not been achieved then explain why not. Then tell me what happens Spoi…

Yes we know the current LLMs cannot solve the original prompt. That's why I experimented with different prompts. The instructions are prompting it to proceed rigorously, as it is a logical problem, not a natural language problem. These models are primarily trained for solving natural language processing tasks, and so they are predisposed to answer in a certain way through training and tuning. The models produce less…

> The instructions are prompting it to proceed rigorously, as it is a logical problem, not a natural language problem.

I think you're missing a bit here. Look at the middle tweet where the person constructed it fail the logic. There are no tricks. What you're missing is the signal you're giving it, how it is spoiling the question in a subtle way. That's very different that a reasoning machine. We can't trust it to reason if it can only "reason" when we give it explicit instructions to do so that do not generalize for many tasks. That's not really reasoning...

> OpenAI could train or tune ChatGPT to "spoil" itself by default

They have and it's provable

Re: LLMs can't do probability

#178

Post by xcv123 »

godelski wrote: post #177
xcv123 wrote: post #173

Earlier quoted context omitted.

Yes we know the current LLMs cannot solve the original prompt. That's why I experimented with different prompts. The instructions are prompting it to proceed rigorously, as it is a logical problem, not a natural language problem. These models are primarily trained for solving natural language processing tasks, and so they are predisposed to answer in a certain way through training and tuning. The models produce less…

> The instructions are prompting it to proceed rigorously, as it is a logical problem, not a natural language problem. I think you're missing a bit here. Look at the middle tweet where the person constructed it fail the logic. There are no tricks. What you're missing is the signal you're giving it, how it is spoiling the question in a subtle way. That's very different that a reasoning machine. We can't trust it to re…

> That's not really reasoning

They are trained a certain way to perform specific types of tasks, primarily natural language processing tasks. They have necessarily learned some methods of reasoning in order to do what they were trained to do. No one is pretending that these are symbolic logic mechanical theorem provers. They are tuned a certain way to respond in a specific manner and they only do what they are told. If you want it to use reasoning then you need to tell it to use reasoning. It's a chat bot running on a neural network and it is not self aware.

Hopefully the next generation of AI will be more reasonable. We are not there yet.

Re: LLMs can't do probability

#179

Post by hervature »

xcv123 wrote: post #175
hervature wrote: post #166

Earlier quoted context omitted.

I would tend to agree with seizethecheese. I don't think anyone says "Markov chain doing naive next token prediction without any semantic model " emphasis on the last part of that sentence. The idiot only says the first part and disregards the last part. The genius says the first part with an understanding that, no matter how complex a semantic model is, the "reasoning" that occurs is completely a function of the tra…

> I think it's clear from this example that the LLM has 0 ability to reason. It's not a 0 or 1. You are oversimplifying it. Obviously neural networks can learn to generalize patterns of reasoning inferred from their training data. We know that they are not using explicitly defined formal systems of reasoning, and they have some limitations compared to those systems. Anyone who seriously studied neural networks or mac…

It actually is 0 or 1 in this case. You either have the power of deduction or you do not. You have either proved a theorem or you have not. If you got to a correct conclusion through incorrect means, you have incorrectly reasoned. There is no spectrum in reasoning. Perhaps a spectrum in abilities across humans but not in the logic itself.

> By the same logic, practically every human on Earth has "0 ability to reason" as their biological neural network will get confused and make mistakes.

And therein lies the problem with this whole debate. I think a huge part of the debate is conflating the fact that most humans do not reason well (I wouldn't say they cannot reason) and make mistakes means that reasoning is something fuzzy and make statements like "LLMs reason about as well as humans". Very few humans outside of mathematicians practice logic on a daily basis. Most humans get by with muscle memory and pattern recognition of previous tasks. Just because LLMs are roughly as good as humans at this behavior does not make them able to reason. I would be totally fine if people just replaced "can reason" with "are useful" within their statements so they would look more like "LLMs are as useful as humans in answering MCAT tests." To imply there is a rational actor deriving responses from first order logic is disingenuous in my opinion.

Re: LLMs can't do probability

#180

Post by krainboltgreene »

xcv123 wrote: post #175
hervature wrote: post #166

Earlier quoted context omitted.

I would tend to agree with seizethecheese. I don't think anyone says "Markov chain doing naive next token prediction without any semantic model " emphasis on the last part of that sentence. The idiot only says the first part and disregards the last part. The genius says the first part with an understanding that, no matter how complex a semantic model is, the "reasoning" that occurs is completely a function of the tra…

> I think it's clear from this example that the LLM has 0 ability to reason. It's not a 0 or 1. You are oversimplifying it. Obviously neural networks can learn to generalize patterns of reasoning inferred from their training data. We know that they are not using explicitly defined formal systems of reasoning, and they have some limitations compared to those systems. Anyone who seriously studied neural networks or mac…

Anyone who has studied neural networks also knows there's no comparison between computer neural networks and human biological neural networks. The name was picked because of a passing familiarity with the biological by someone who didn't have any experience in biological neural networks. It's been sufficiently proven they have no similarity by countless academics.
Post reply on HN