Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

LLMs can't do probability

brainsteam.co.uk

Post reply on HN
Print view View original

141–150 of 211 posts

Re: LLMs can't do probability

#141

Post by xcv123 »

seizethecheese wrote: post #130
panarky wrote: post #129

Earlier quoted context omitted.

Your understanding of how LLMs work is overly simplistic and incomplete. Yes, doing probabilistic next-word prediction plays a role in how LLMs generate text output, but that's not the whole story. LLMs "understand" (to a degree): They develop complex internal representations of concepts they've been trained on. This isn't just about word association; they develop an understanding of the relationships between objects…

I feel like this is the midwit meme response and the idiot and genius are just saying “it’s next token probabilistic geberation”

A typical mid-wit response is to say it's just a Markov chain doing naive next token prediction without any semantic model. That's not how deep learning works.

The LLM neural network contains a semantic model and it performs some type of reasoning over that model. The idiot and the genius both can see that ChatGPT has some reasoning capability.

https://www.cs.toronto.edu/~hinton/absps/AIJmapping.pdf

"This 1990 paper demonstrated how neural networks could learn to represent and reason about part-whole hierarchical relationships, using family trees as the example domain.

By training on examples of family relations like parent-child and grandparent-grandchild, the neural network was able to capture the underlying logical patterns and reason about new family tree instances not seen during training.

This seminal work highlighted that neural networks can go beyond just memorizing training examples, and instead learn abstract representations that enable reasoning and generalization"

Re: LLMs can't do probability

#142

Post by getoj »

sqeaky wrote: post #136
mk_stjames wrote: post #125

Earlier quoted context omitted.

But it it like asking a person for them to generate the same thing, but when the start to list off their answers stopping them by throwing up your hand after their first response, writing that down, and then going back in time and asking that person to do the same thing, and stopping them again, and repeat- and then being surprised after 1000 times that you didn't get a reasonable distribution. Meanwhile if you let t…

If You asked me to pick a random number between one and six and ignore all previous attempts, I would roll a die and you would get a uniform distribution (or at least not 99% the same number). If you are saying that this thing can't generate random numbers on the first try then it can't generate random numbers. Which makes sense. Computers have a really hard time with random, and that's why every computer science cou…

>If You asked me to pick a random number between one and six and ignore all previous attempts, I would roll a die and you would get a uniform distribution

I believe what GP is getting at is that if you didn't have a die, and you truly ignored all your previous attempts to the point of genuinely forgetting that the question had been asked, then your answer would likely be the same every time. Imagine asking a person with severe Alzheimer's to pick a number, then asking again a few minutes later. You'd probably get the same answer.

Re: LLMs can't do probability

#143

Post by zrezzed »

abdullahkhalids wrote: post #128
mk_stjames wrote: post #122

I went through all the comments here and I'm still not seeing anyone address this: If I am reading this person correctly... they prompted the model with the prompt directly 1000 times... but only for the first time. They did not allow the model to actually run a context for chat. Simply, output the first in a list of 'left' and 'right' and favor 'left' 80% of the time... but then the author only asked for the first o…

Isn't this way of prompting roughly equal to asking a 1000 people to pick left or right with 80% prob of left? I imagine, the result with humans will be closer to 80:20 than whatever happened with the LLM.

I agree it’s equivalent, and that’s a great way to think about it.

But… I wouldn’t be surprised if humans answered closer to the LLM results than 80:20. I’d actually be surprised if humans converged very close to the right result.

Would be a fun mechanical Turk experiment to run.

Re: LLMs can't do probability

#144

Post by xcv123 »

SrslyJosh wrote: post #139
panarky wrote: post #129

Earlier quoted context omitted.

Your understanding of how LLMs work is overly simplistic and incomplete. Yes, doing probabilistic next-word prediction plays a role in how LLMs generate text output, but that's not the whole story. LLMs "understand" (to a degree): They develop complex internal representations of concepts they've been trained on. This isn't just about word association; they develop an understanding of the relationships between objects…

That's a lot of anthropomorphizing, but I don't see anything to back up your claims.

Neural Networks have been studied for a long time. We learned this in 1990.

https://news.ycombinator.com/item?id=40230764

Re: LLMs can't do probability

#145

Post by richardw »

panarky wrote: post #129
Terr_ wrote: post #115

My rule of thumb is to take every single LLM prompt and just imagine that it's prefixed with: "Computer, focus on generating output that resembles the words people in the past used after they were given the following words..."

Your understanding of how LLMs work is overly simplistic and incomplete. Yes, doing probabilistic next-word prediction plays a role in how LLMs generate text output, but that's not the whole story. LLMs "understand" (to a degree): They develop complex internal representations of concepts they've been trained on. This isn't just about word association; they develop an understanding of the relationships between objects…

I know that's the messaging, but the real link to reality is very tenuous. This was a great example from the last couple days:

https://twitter.com/colin_fraser/status/1785132544482226679

I just tried a similar question now with ChatGPT4:

"If a man and a goat are on one side of a river, what is the minimum amount of trips required to get the man and goat to the other side in a boat. Assume the boat can hold at most one animal and one human."

ChatGPT: 3 trips

That is very much closer to "trying to predict next word from examples" than "billion-dollar model with internal reasoning".

Re: LLMs can't do probability

#146

Post by devjab »

mk_stjames wrote: post #122

I went through all the comments here and I'm still not seeing anyone address this: If I am reading this person correctly... they prompted the model with the prompt directly 1000 times... but only for the first time. They did not allow the model to actually run a context for chat. Simply, output the first in a list of 'left' and 'right' and favor 'left' 80% of the time... but then the author only asked for the first o…

> The model takes the prompt. The first following output token it chooses, for this specfic model, happens to be 'Left'. They shut down the prompt and prompt again. Of course the next output will be 'Left'.

I think what you say is correct, but isn’t this exactly the point the author is trying to make? You explain why it happens, but ultimately the result is still that the LLMs can’t do probability. At least not in the way the author presents it.

What I’m more interested in is what sort of consequences these debates and different understandings/views of LLMs will have on us in general. We monitor the usage of co-pilot in our organisation and it’s been interesting to watch how a lot of employees have started promoting the same task to multiple “agents” simultaneously. One person tends to always run at least five at once. So naturally we dug a little into why people were doing it, and, they are doing it because “it’s the fastest way to get something useful”. Which is probably not too surprising to a lot of you, but it was sort of hilarious to follow along and see how two promoted “agents” gave completely conflicting answers.

I know LLMs will probably get better at being lucky, but some of these tasks… honestly we had a couple of our juniors who were very you’re of always going GPT of co-pilot first, simply look at the official documentation and we timed them, and it was so much faster for them to not use LLMs. This isn’t me saying LLMs suck, we deliberately did it to make sure the juniors in question had an “oh” moment. But it’s interesting to see how quickly our employees have adopted LLMs.

Re: LLMs can't do probability

#147

Post by retrac »

SrslyJosh wrote: post #139
panarky wrote: post #129

Earlier quoted context omitted.

Your understanding of how LLMs work is overly simplistic and incomplete. Yes, doing probabilistic next-word prediction plays a role in how LLMs generate text output, but that's not the whole story. LLMs "understand" (to a degree): They develop complex internal representations of concepts they've been trained on. This isn't just about word association; they develop an understanding of the relationships between objects…

That's a lot of anthropomorphizing, but I don't see anything to back up your claims.

I think it's still more in the realm of philosophy. But I do have an argument that NNs demonstrate abstract, generalized learning: the transfer learning effect.

Neural networks pre-trained on data for a completely different task, learn new tasks much faster. With a GPT-like transformer, you can feed it PCM audio samples encoded as uuencoded text, or paintings encoded in the same way, and it learns how to translate English Russian when later trained on that, much faster than from a completely randomized model. There's something common to those seemingly disparate tasks that is learned. "Abstraction" may be the right word for this.

Re: LLMs can't do probability

#148

Post by zrezzed »

mk_stjames wrote: post #122

I went through all the comments here and I'm still not seeing anyone address this: If I am reading this person correctly... they prompted the model with the prompt directly 1000 times... but only for the first time. They did not allow the model to actually run a context for chat. Simply, output the first in a list of 'left' and 'right' and favor 'left' 80% of the time... but then the author only asked for the first o…

I’d bet the author does understand.

I’m fairly certain subsequent runs even with the same prompt will make selections with different entropy. Even with low temperature, there’s no reason a good enough model couldn’t start such a list with “Right” 20% of the time.

Re: LLMs can't do probability

#149

Post by buescher »

gwern wrote: post #81
robertclaus wrote: post #54

I wonder if you could actually fine tune an LLM to do better on this. As some of the comments point out, the issue here is that the possible output probabilities combined with the model temperature don't actually result in the probabilities requested in the prompt. If you trained on specific generated data with real distributions would it learn to compensate appropriately? Would that carry over to novel probability p…

> If you trained on specific generated data with real distributions It was trained on generated data from real distributions! The datasets LLMs are trained on include gigabytes of real data from real distributions, in addition to all of the code/stats/etc samples. The question you should be asking is 'why did it stop being able to predict real distributions?' And we already know the answer: RLHF. https://news.ycombin…

Do we know in any detail who provided the RLHF and according to what rules for any of these models?

Re: LLMs can't do probability

#150

Post by IIAOPSW »

apendleton wrote: post #124
mk_stjames wrote: post #122

I went through all the comments here and I'm still not seeing anyone address this: If I am reading this person correctly... they prompted the model with the prompt directly 1000 times... but only for the first time. They did not allow the model to actually run a context for chat. Simply, output the first in a list of 'left' and 'right' and favor 'left' 80% of the time... but then the author only asked for the first o…

I think this is overthinking it. ChatGPT is billed as a general-purpose question-answerer, as are its competitors. A regular user shouldn't have to care how it works, or know anything about context or temperature or whatever. They ask a question, it answers and appears to have given a plausible answer, but doesn't actually do the task that was asked for and that it appears to do. What the technical reasons are that i…

I'm just impressed that it answered "left" rather than outputting python code that I could run which would sample the list ["left", "right"] with an 80% bias.
Post reply on HN