Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

LLMs can't do probability

brainsteam.co.uk

Post reply on HN
Print view View original

181–190 of 211 posts

Re: LLMs can't do probability

#181

Post by xcv123 »

hervature wrote: post #179
xcv123 wrote: post #175

Earlier quoted context omitted.

> I think it's clear from this example that the LLM has 0 ability to reason. It's not a 0 or 1. You are oversimplifying it. Obviously neural networks can learn to generalize patterns of reasoning inferred from their training data. We know that they are not using explicitly defined formal systems of reasoning, and they have some limitations compared to those systems. Anyone who seriously studied neural networks or mac…

It actually is 0 or 1 in this case. You either have the power of deduction or you do not. You have either proved a theorem or you have not. If you got to a correct conclusion through incorrect means, you have incorrectly reasoned. There is no spectrum in reasoning. Perhaps a spectrum in abilities across humans but not in the logic itself. > By the same logic, practically every human on Earth has "0 ability to reason"…

Only if you define reasoning ability as exactly equivalent in capability to a formal theorem prover. But that is a difference in tribe or philosophy. Your Symbolic/Classical rule-based AI tribe versus the Connectionist AI tribe. No point discussing further as it's like arguing Democrat vs Republican. Both approaches have their strengths and weaknesses.

Re: LLMs can't do probability

#182

Post by sam0x17 »

richardw wrote: post #159
xcv123 wrote: post #155

Earlier quoted context omitted.

The LLM does figure it out if you ask further questions in the same chat. Here's GPT-3.5 https://chat.openai.com/share/a8669390-8eb0-46c2-b804-3aafc3...

I tried "are you sure", which often triggers some reasoning, and it was pretty confident. I'm trying not to give it the answer, but run it as if I didn't have any special knowledge. GPT + human > GPT. I mean, we're treating these things like another kind of intelligence, not a hammer. GPT4: https://chat.openai.com/share/1beb5391-d321-4515-945e-38233f...

Another really oddly effective one is offering it a $500 tip

Re: LLMs can't do probability

#183

Post by xcv123 »

krainboltgreene wrote: post #180
xcv123 wrote: post #175

Earlier quoted context omitted.

> I think it's clear from this example that the LLM has 0 ability to reason. It's not a 0 or 1. You are oversimplifying it. Obviously neural networks can learn to generalize patterns of reasoning inferred from their training data. We know that they are not using explicitly defined formal systems of reasoning, and they have some limitations compared to those systems. Anyone who seriously studied neural networks or mac…

Anyone who has studied neural networks also knows there's no comparison between computer neural networks and human biological neural networks. The name was picked because of a passing familiarity with the biological by someone who didn't have any experience in biological neural networks. It's been sufficiently proven they have no similarity by countless academics.

That is a blatant oversimplification and not true. There are both similarities and differences. New ANN training methods are inspired by studies of biological neural networks (Dropout Regularization is one example)

Re: LLMs can't do probability

#184

Post by hervature »

xcv123 wrote: post #181
hervature wrote: post #179

Earlier quoted context omitted.

It actually is 0 or 1 in this case. You either have the power of deduction or you do not. You have either proved a theorem or you have not. If you got to a correct conclusion through incorrect means, you have incorrectly reasoned. There is no spectrum in reasoning. Perhaps a spectrum in abilities across humans but not in the logic itself. > By the same logic, practically every human on Earth has "0 ability to reason"…

Only if you define reasoning ability as exactly equivalent in capability to a formal theorem prover. But that is a difference in tribe or philosophy. Your Symbolic/Classical rule-based AI tribe versus the Connectionist AI tribe. No point discussing further as it's like arguing Democrat vs Republican. Both approaches have their strengths and weaknesses.

I am defining reason exactly as Wikipedia puts it: "Reason is the capacity of applying logic consciously by drawing conclusions from new or existing information, with the aim of seeking the truth."

There are no tribes here. Republican vs. Democrat, I do not care. If your logic is unsound, I'm going to call you out even if I agree with the conclusion. State your definitions so we can have a formal logic-based debate. For the record, I use neural networks every day and believe that they are incredibly useful and can be purpose-built to beat humans on a large set of tasks. Can they reason? No. Can formal theorem provers reason? No they cannot. They can only verify.

Re: LLMs can't do probability

#185

Post by xcv123 »

hervature wrote: post #184
xcv123 wrote: post #181

Earlier quoted context omitted.

Only if you define reasoning ability as exactly equivalent in capability to a formal theorem prover. But that is a difference in tribe or philosophy. Your Symbolic/Classical rule-based AI tribe versus the Connectionist AI tribe. No point discussing further as it's like arguing Democrat vs Republican. Both approaches have their strengths and weaknesses.

I am defining reason exactly as Wikipedia puts it: "Reason is the capacity of applying logic consciously by drawing conclusions from new or existing information, with the aim of seeking the truth." There are no tribes here. Republican vs. Democrat, I do not care. If your logic is unsound, I'm going to call you out even if I agree with the conclusion. State your definitions so we can have a formal logic-based debate.…

LLMs have already demonstrated their ability to generally apply abstract rules of logical reasoning to problems outside of their training set. Those are instances of reasoning.

The models are learned from data, and they infer logical reasoning methods implied in the data, but they are not explicitly rules driven and so they produce output which may be logically inconsistent.

Solving natural language processing tasks requires an understanding of the context of words, and this contextual understanding is constructed through a form of reasoning. You can choose to ignore all of the reasoning that they are demonstrably capable of, and narrowly focus on their imperfections, then declare they are capable of "0 reasoning", but that conclusion is an oversimplification. With actual "0 reasoning" ability they wouldn't be able to perform standard NLP tasks such as translation or summarization anywhere close to their current level of accuracy.

The Wikipedia definition refers to consciousness, so that excludes machines. That informal definition is not useful in the context of AI. We need to use technical definitions. Otherwise it is subjective and we end up wasting time squabbling about semantics.

Re: LLMs can't do probability

#186

Post by bamboozled »

inopinatus wrote: post #123
Terr_ wrote: post #115

My rule of thumb is to take every single LLM prompt and just imagine that it's prefixed with: "Computer, focus on generating output that resembles the words people in the past used after they were given the following words..."

Correct. You must perceive them as plausibility engines. The unstated hypothesis is that plausibility of output may converge towards correctness of output with increasing scale and sophistication. This hypothesis remains very far from proven.

I also think you get the best results when thinking about using them this way too, any other way of using them seems to end in disappointment.

Re: LLMs can't do probability

#187

Post by 6gvONxR4sf7o »

intended wrote: post #110
6gvONxR4sf7o wrote: post #62

Earlier quoted context omitted.

Nope, because all of that is taken care of by the mechanisms for evaluating the model. Strictly speaking, the model outputs a probability distribution. The question is why that distribution doesn’t match the instructions.

I think I maybe get where you are coming from, but still how? I feel we are discussing 2 different use cases. 1) Prompt 1: “ You are a weighted random choice generator. About 80% of the time please say ‘left’ and about 20% of the time say ‘right’. Simply reply with left or right. Do not say anything else" ” 2) Assume that the training data gives examples of 2.1) single coin flips 2.2) multiple coin flips Consider a s…

Nope, i’m describing prompt 1. The output of the model is a distribution over tokens, which is then sampled by the system to get the next word/token. This is what people mean when they talk about the logits in these models. So if you ask it for one single sample, you hope it might give you a logit vector that corresponds to 80% ‘left’ token and 20% ‘right’ token (assuming those words are single tokens, but it all still works otherwise, but is less concise to explain). When it then autocompletes, it samples from that distribution (a weighted coin flip, in your terms).

So it’s just neat that the weights in the coin flip don’t match what is asked for.

Re: LLMs can't do probability

#188

Post by hervature »

xcv123 wrote: post #185
hervature wrote: post #184

Earlier quoted context omitted.

I am defining reason exactly as Wikipedia puts it: "Reason is the capacity of applying logic consciously by drawing conclusions from new or existing information, with the aim of seeking the truth." There are no tribes here. Republican vs. Democrat, I do not care. If your logic is unsound, I'm going to call you out even if I agree with the conclusion. State your definitions so we can have a formal logic-based debate.…

LLMs have already demonstrated their ability to generally apply abstract rules of logical reasoning to problems outside of their training set. Those are instances of reasoning. The models are learned from data, and they infer logical reasoning methods implied in the data, but they are not explicitly rules driven and so they produce output which may be logically inconsistent. Solving natural language processing tasks…

> We need to use technical definitions.

I agree. I more or less disagree with all your other claims. A linear model is not reasoning when it extrapolates. Neural networks do not "apply abstract rules of logical reasoning to problems outside of their training set".

> Solving natural language processing tasks requires an understanding of the context of words

Agree.

> and this contextual understanding is constructed through a form of reasoning

Disagree. This models are predictive models. Here is a simple estimator for your height: ((mom's height)+(dad's height))/2. This estimator is very good and modern genomic techniques struggle to beat it. Does this model reason? Obviously not. Yet, the 1B parameter genomics models do reason about the genes? That makes no sense. They are both models. Incapable of reason.

> You can choose to ignore all of the reasoning that they are demonstrably capable of

They have not demonstrated any reasoning. Just because they can solve olympiad level math questions, it does not mean they can reason. This does not take away from the power of these models. These models will have full conversations about reasoning, explain syllogistic logic, and apply different deductions which seemingly show they are reasoning. It's all impressive. Then you are harshly reminded by the following interaction that it is just an amazing ability to memorize:

Again with Claude. Prompt:

A farmer and a goat need to cross a river. There is a boat that can fit exactly one human and one animal. What is the least number of crossings needed to get everything to the other side of the river?

===

Okay, let's approach this classic logic puzzle systematically:

Given:

- There is a farmer, a goat, and a boat that can carry one human and one animal at a time.

- The farmer cannot leave the goat alone with the crop, as the goat will eat the crop.

- The objective is to get the farmer, goat, and the boat across the river using the minimum number of crossings.

To solve this, we need to find a sequence of crossings that adheres to the constraint and achieves the objective in the fewest steps.

Here's one valid solution that requires 7 crossings:

1) Farmer rows the goat across the river.

2) Farmer rows back alone.

3) Farmer rows the crop across the river.

4) Farmer rows the goat back.

5) Farmer rows across again.

6) Goat rows across.

7) Farmer rows the goat across.

Explanation:

- Crossing 1 gets the goat across initially.

- Crossing 2 brings the farmer back to pick up the crop.

- Crossing 3 takes the crop across.

- Crossing 4 brings the goat back so it's not alone with the crop.

- Crossing 5 takes the farmer across.

- Crossing 6 takes the goat across since it's now safe.

- Crossing 7 takes the farmer and goat together to the desired side.

This sequence ensures the goat is never left unattended with the crop, adhering to the constraint.

It can be proven that 7 crossings is the minimum required, as:

- The goat must cross twice (there and back).

- The crop must cross once.

- The farmer must cross at least twice to accompany the goat and crop.

- Additional crossings are needed to reposition the boat.

So the optimal solution involves 2 crossings for the goat, 1 for the crop, and a minimum of 4 additional crossings for the farmer and boat repositioning, totaling 7 crossings.

Therefore, the least number of crossings needed to get the farmer, goat, and boat across the river is 7.

===

All appearances of reasoning are nullified by a response that pulls stuff out of thin air and states it confidently.

Re: LLMs can't do probability

#189

Post by sqeaky »

getoj wrote: post #142
sqeaky wrote: post #136

Earlier quoted context omitted.

If You asked me to pick a random number between one and six and ignore all previous attempts, I would roll a die and you would get a uniform distribution (or at least not 99% the same number). If you are saying that this thing can't generate random numbers on the first try then it can't generate random numbers. Which makes sense. Computers have a really hard time with random, and that's why every computer science cou…

>If You asked me to pick a random number between one and six and ignore all previous attempts, I would roll a die and you would get a uniform distribution I believe what GP is getting at is that if you didn't have a die, and you truly ignored all your previous attempts to the point of genuinely forgetting that the question had been asked, then your answer would likely be the same every time. Imagine asking a person w…

Yeah, I get what they are saying and there's no reason to believe that.

The die is an analogy for our decision making. They are implicitly claiming that randomness must come in an order and that simply isn't how randomness works or even halfway decent pseudo randomness.

Any system whether it be a die, a person, or an llm doesn't have to know about its previous random choices to make random choices that follow some distribution going forward presuming it's actually capable of randomness.

Re: LLMs can't do probability

#190

Post by xcv123 »

hervature wrote: post #188
xcv123 wrote: post #185

Earlier quoted context omitted.

LLMs have already demonstrated their ability to generally apply abstract rules of logical reasoning to problems outside of their training set. Those are instances of reasoning. The models are learned from data, and they infer logical reasoning methods implied in the data, but they are not explicitly rules driven and so they produce output which may be logically inconsistent. Solving natural language processing tasks…

> We need to use technical definitions. I agree. I more or less disagree with all your other claims. A linear model is not reasoning when it extrapolates. Neural networks do not "apply abstract rules of logical reasoning to problems outside of their training set". > Solving natural language processing tasks requires an understanding of the context of words Agree. > and this contextual understanding is constructed thr…

I'll refer you back to this 1990 paper by Geoffrey Hinton. Up to you if you want to investigate this further.

If you can prove this is wrong then you should publish your result.

https://www.cs.toronto.edu/~hinton/absps/AIJmapping.pdf

"This 1990 paper demonstrated how neural networks could learn to represent and reason about part-whole hierarchical relationships, using family trees as the example domain.

By training on examples of family relations like parent-child and grandparent-grandchild, the neural network was able to capture the underlying logical patterns and reason about new family tree instances not seen during training.

This seminal work highlighted that neural networks can go beyond just memorizing training examples, and instead learn abstract representations that enable reasoning and generalization"

Post reply on HN