Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Simple tasks showing reasoning breakdown in state-of-the-art LLMs

arxiv.org

Post reply on HN
Print view View original

321–330 of 393 posts

Re: Simple tasks showing reasoning breakdown in state-of-the-art LLMs

#321

Post by jononomo »

naasking wrote: post #310
PheonixPharts wrote: post #102

Earlier quoted context omitted.

People keep saying it because that's literally how LLMs work. They run Montecarlo sampling over a very impressive latent linguistic space. These models are not fundamentally different than the Markov chains of yore except that these latent representations are incredibly powerful. We haven't even started to approach the largest problem which is moving beyond what is essentially a greedy token level search of this ling…

> LLMs are not reasoning machines. They are basically semantic compression machines with a build in search feature. This is just a god of the gaps argument. Understanding is a form of semantic compression. So you're saying we have a system that can learn and construct a database of semantic information, then search it and compose novel, structured and coherent semantic content to respond to an a priori unknown prompt…

I bite the bullet on the god of the gaps

Re: Simple tasks showing reasoning breakdown in state-of-the-art LLMs

#323

Post by TeMPOraL »

tsunamifury wrote: post #90
pawelmurias wrote: post #59

Earlier quoted context omitted.

LLMs are idiots. They can't reason properly and only parrot stuff https://chatgpt.com/share/dcb4ff4e-e8a2-463b-86ec-9caf10b6e6... Sometimes they get the answer right to something really complex because it fits a pattern, but sometimes they answer with something really really stupid.

Why are so many people so insistent on saying this? I’m guessing you are in denial that we can make a simulated reasoning machine?

There's some irony in seeing people parrot the argument that LLMs are parrots.

Re: Simple tasks showing reasoning breakdown in state-of-the-art LLMs

#324

Post by photon_lines »

mcguire wrote: post #289
photon_lines wrote: post #80

Earlier quoted context omitted.

I'm not sure what you mean by 'it will not scale well.' When we humans learn that we make a mistake - we make a note and we hold the correct answer in memory - the next time we're prompted with a similar prompt, we can use our old memories to come up with the correct solution. I just did a simple test for this same exact problem using ChatGPT 3.5: 'Can you reformulate the following problem using Prolog? When you exec…

" My prompt after ChatGPT gave me an answer of 5: 'Yes but you aren't including Alice in your logic. Shouldn't the correct answer be 6 (since Alice is also a sister)?' " Useful, if you know what the answer is. What happens if you don't give it the correct answer?

What happens when you don't give a human being the correct answer? :)

Re: Simple tasks showing reasoning breakdown in state-of-the-art LLMs

#325

Post by photon_lines »

pbhjpbhj wrote: post #168
photon_lines wrote: post #80

Earlier quoted context omitted.

I'm not sure what you mean by 'it will not scale well.' When we humans learn that we make a mistake - we make a note and we hold the correct answer in memory - the next time we're prompted with a similar prompt, we can use our old memories to come up with the correct solution. I just did a simple test for this same exact problem using ChatGPT 3.5: 'Can you reformulate the following problem using Prolog? When you exec…

When you ask again the prompt includes the context of your previous question and correction. When I ask the prompt doesn't have that context so the model fails to give me the correct answer. I'm using the default free model in the app, based on GPT4.

Yup - well this is where my suggestion is to change the GPT architecture. You can think of having a logic program function as the 'frontal lobe' of the general pre-trained auto-associative model. This 'frontal lobe' region would try to come up with logical sequences to go along with it's internal auto-associative representations. Of course - the logic programming piece is just one approach - maybe chain of though or chain of reason prompting could work here too as many humans I think use this chain-of-reasoning approach themselves. Logic programming to me would function as a suggested shortcut.

Re: Simple tasks showing reasoning breakdown in state-of-the-art LLMs

#326

Post by TeMPOraL »

imtringued wrote: post #137
tsunamifury wrote: post #90

Earlier quoted context omitted.

Why are so many people so insistent on saying this? I’m guessing you are in denial that we can make a simulated reasoning machine?

No it is because supervised and self supervised learning happen to produce reasoning as a byproduct. For some reason people think that telling a model to recite a trillion tokens somehow will improve it beyond the recitation of those tokens. I mean, in theory you can select the training data so that it will learn what you want, but then again you are limited to what you taught it directly. The problem is that these m…

> The problem is that these models weren't trained to reason.

Except they kind of were. Specifically, they were trained to predict next tokens based on text input, with the optimization function being, does the result make sense to a human?. That's embedded in the training data: it's not random strings, it's output of human reasoning, both basic and sophisticated. That's also what RLHF selects for later on. The models are indeed forced to simulate reasoning.

> don't train it to do something else and then expect it to do the thing you didn't train it for.

That's the difference between AGI and specialized AI - AGI is supposed to do the things you didn't train it to do.

Re: Simple tasks showing reasoning breakdown in state-of-the-art LLMs

#327

Post by HarHarVeryFunny »

An LLM is a stack of transformer layers (N=96 for GPT-2). Each word/token output is the result of exactly N sentence transformations from input to output.

Clearly there are many things (specifically things requiring deliberation and planning) that require an open ended number of thought steps, and therefore could only, even in theory, we done by an LLM in "think step by step" fashion where it breaks the problem down into pieces (maybe recursively) each simple enough to be done in In practice it seems that LLM's ability (esp. given their lack of any internal working memory) to maintain long term focus for an extended task is poor, so the theoretical benefit of using their own memory as a scratchpad for planning doesn't seem to be realized.

The notion of scale it up, and/or train it right, and a transformer can implement AGI, seems to be more wishful thinking than based on any analysis of what the architecture is and is not capable of. The lack of support for continuous incremental learning seems to be a deal-breaker for intelligence that people just ignore.

Re: Simple tasks showing reasoning breakdown in state-of-the-art LLMs

#328

Post by Vegenoid »

solidasparagus wrote: post #302
throwanem wrote: post #288

Earlier quoted context omitted.

Responding to an unsubstantiated claim with an unsubstantiated claim just adds another unsubstantiated claim. So far as I know, whether the universe behaves deterministically remains an unsolved question. Given that, your statement here would already be one of belief rather than fact, even before we get to the parentheticals. There is information here, but not about whether LLMs can develop into AGI.

Fine, you can ignore my previous comment, that's just my answer to the question that this discussion ultimately takes you to. But I feel like you are just sitting on the sidelines making strawmen and playing pedantic games instead of saying anything constructive. The original comment said: > If you really think about what an LLM is you would think there is no way that leads to general purpose AI. This is an inflammat…

> I made no claims, I asked no one to prove anything wrong.

Your original comment was:

> It is an autoregressive sequence predictor/generator. Explain to me how humans are fundamentally different.

Which would be interpreted by most reasonable people as you making the claim that humans are autoregressive sequence predictors, and asking people to prove you wrong. I can see how you could say this without intending to make that claim, but most people will interpret this as you making that claim.

I do not intend to inflame things or discredit what you are saying, but just to say that if you did not intend to make a claim or ask people to prove you wrong, a different approach would be more successful in the future.

Re: Simple tasks showing reasoning breakdown in state-of-the-art LLMs

#329

Post by bongodongobob »

hellojesus wrote: post #241
bongodongobob wrote: post #218

Earlier quoted context omitted.

Eat something and take a nap, you sound unhinged. ChatGPT has nearly doubled my work output, most of my job is system admin infra type stuff and it's ridiculously good at troubleshooting odd issues. Hopefully you can find a use case for it someday, until then, the rest of us will continue to be more productive.

Do you have any concern about the data you're feeding to the vendor serving your prompts? I've had junior devs tell me they use chatgippity to combine excel workbooks, and when I confirm they're not self hosting a llm to do it, I ask if they think it's a good idea to hand over company data to openai. They don't care. In a world of tight security, I find it astonishing that so many people willingly give away trade sec…

I do think it's fine yes, but I also am not giving it anything sensitive.

Re: Simple tasks showing reasoning breakdown in state-of-the-art LLMs

#330

Post by elicksaur »

tsunamifury wrote: post #90
pawelmurias wrote: post #59

Earlier quoted context omitted.

LLMs are idiots. They can't reason properly and only parrot stuff https://chatgpt.com/share/dcb4ff4e-e8a2-463b-86ec-9caf10b6e6... Sometimes they get the answer right to something really complex because it fits a pattern, but sometimes they answer with something really really stupid.

Why are so many people so insistent on saying this? I’m guessing you are in denial that we can make a simulated reasoning machine?

Maybe people have different experiences with the products than you.

A simulated reasoning machine being possible does not mean that current LLMs are simulated thinking machines.

Maybe you should try asking chatgpt for advice on how to understand other people’s perspectives: https://chatgpt.com/share/3d63c646-859b-4903-897e-9a0cb7e47b...

Post reply on HN