Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

What can LLMs never do?

strangeloopcanon.com

Post reply on HN
Print view View original

151–160 of 385 posts

Re: What can LLMs never do?

#151

Post by ianbicking »

cs702 wrote: post #14

Fantastic essay. Highly recommended! I agree with all key points: * There are problems that are easy for human beings but hard for current LLMs (and maybe impossible for them; no one knows). Examples include playing Wordle and predicting cellular automata (including Turing-complete ones like Rule 110). We don't fully understand why current LLMs are bad at these tasks. * Providing an LLM with examples and step-by-step…

"Providing an LLM with examples and step-by-step instructions in a prompt means the user is figuring out the "reasoning steps" and handing them to the LLM, instead of the LLM figuring them out by itself. We have "reasoning machines" that are intelligent but seem to be hitting fundamental limits we don't understand."

One thing an LLM _also_ doesn't bring to the table is an opinion. We can push it in that direction by giving it a role ("you are an expert developer" etc), but it's a bit weak.

If you give an LLM an easy task with minimal instructions it will do the task in the most conventional, common sense fashion. And why shouldn't it? It has no opinion, your prompt doesn't give it an opinion, so it just does the most normal-seeming thing. If you want it to solve the task in any other way then you have to tell it to do so.

I think a hard task is similar. If you don't tell the LLM _how_ to solve the hard task then it will try to approach it in the most conventional, common sense way. Instead of just boring results for a hard task the result is often failure. But hard problems approached with conventional common sense will often result in failures! Giving the LLM a thought process to follow is a quick education on how to solve the problem.

Maybe we just need to train the LLM on more problem solving? And maybe LLMs worked better when they were initially trained on code for exactly that reason, it's a much larger corpus of task-solving examples than is available elsewhere. That is, maybe we don't talk often enough and clearly enough about how to solve natural language problems in order for the models to really learn those techniques.

Also, as the author talks about in the article with respect to agents, the inability to rewind responses may keep the LLM from addressing problems in the ways humans do, but that can also be addressed with agents or multi-prompt approaches. These approaches don't seem that impressive in practice right now, but maybe we just need to figure it out (and maybe with better training the models themselves will be better at handling these recursive calls).

Re: What can LLMs never do?

#152

Post by Der_Einzige »

Why did the author have to claim that it's not tokenization issues?

This issue, or at least similar ones, absolutely is due to tokenization issues.

Karpathy is right that nearly every modern problem with LLMs is due to tokenization, but if you don't believe him, maybe see this work by gwern: https://gwern.net/gpt-3#bpes or this work by yours truly: https://aclanthology.org/2022.cai-1.2/

Re: What can LLMs never do?

#153

Post by ricardobeat »

_wire_ wrote: post #127
fallingsquirrel wrote: post #66

Earlier quoted context omitted.

I just tried this locally with llama3-8b and it handled it fine. Claude 3 Sonnet passes your test too, in case you don't have the hardware for a local model. You might want to consider moving on from ChatGPT since their models have been RLHF'd to hell in the name of "safety" and are falling behind in many ways as a result. --- transcript: $ ollama run llama3:8b >>> tell me an interesting fact about etymology Here's o…

Why is any of this "interesting"? Is the response interesting because you don't know it? —How does it know this? Is the response interesting because you do know it or might have offered it yourself? —Confirmation bias. Is it interesting because a lot of training references contextualize it as "interesting?" —Begged question. Is it contextually interesting? —What is the context? A robot refers to robots? How unexpecte…

Pretty sure the point here was Llama3 respecting the command to not mention that this is interesting, not adding filler, rather than the output fact being interesting or not.

Re: What can LLMs never do?

#154

Post by Der_Einzige »

Xenoamorphous wrote: post #18

There’s many things they can’t do. Even a simple rule like “ensure that numbers from one to ten are written as words and numbers greater ten as digits in the given text” fails for me for so many examples even if it works for many others; few shot, chain of thought, many versions of the prompt, it doesn’t matter. Sometimes LLMs will even change the number to something else, even with temp set to 0. And then there’s th…

All of these issues are entirely due to the tokenization scheme. Literally all of them

You could get this behavior implemented perfectly with constrained text gen techniques like grammars or any of the various libraries implementing constrained text gen (i.e. guidance)

Re: What can LLMs never do?

#155

Post by tobiasSoftware »

richardw wrote: post #145
_wire_ wrote: post #119

Earlier quoted context omitted.

>We don't fully understand why current LLMs are bad at these tasks. In complete seriousness, can anyone can explain why LLMs are good at some tasks?

LLM’s are a compressed and lossy form of our combined writing output, which it turns out is similarly structured enough to make new combinations of text seem reasonable, even enough to display simple reasoning. I find it useful to think “what can I expect from speaking with the dataset of combined writing of people”, rather than treating a basic LLM as a mind. That doesn’t mean we won’t end up approximating one event…

"I find it useful to think “what can I expect from speaking with the dataset of combined writing of people”, rather than treating a basic LLM as a mind."

I've been doing this as well, mentally I think of LLMs as the librarians of the internet.

Re: What can LLMs never do?

#156

Post by ricardobeat »

moffkalast wrote: post #138
keskival wrote: post #129

All points described are simple artifacts of tokenization.

> If a model is trained on a sentence of the form "A is B", it will not automatically generalize to the reverse direction "B is A". This is the Reversal Curse. This is not a tokenization artefact. And furthermore it's a problem for human brains as well. Let's say you get a name, idk, Tom Cruise. You immediately know what his face looks like. Now let's say you get a random face. How quickly would you be able to tell m…

That's not a great example. Remembering a face is memory recall, whereas what's at stake here is LLMs not being able to infer simple relationships - if it learns from data that "John owns the red bicycle", it will succeed at answering "what does John own", but not "who owns the red bicycle". The relationship it learns is unidirectional.

Here's the intro to the paper that brought this to light: https://www.lesswrong.com/posts/SCqDipWAhZ49JNdmL/paper-llms...

Re: What can LLMs never do?

#157

Post by Der_Einzige »

donatj wrote: post #62

I have been trying to generate some text recently using the ChatGPT API. No matter how I word “Include any interesting facts or anecdotes without commenting on the fact being interesting” it ALWAYS starts out “One interesting fact about” or similar phrasing. I have honestly spent multiple hours trying to word the prompt so it will stop including introductory phrases and just include the fact straight. I have gone so…

API driven LLMs on purpose don't implement core features which would enable which you want, for example, negative prompting.

You can negative prompt any LLM with stuff like "always write the word interesting in your response".

You can also use techniques for modifying logprobs of tokens, which is avaialble in gpt-4 api (but is hard to use). You can literally ban "interesting" from its vocabulary.

You could even use representation steering techniques to do this using control vectors. See this library as an example: https://github.com/Hellisotherpeople/llm_steer-oobabooga

Re: What can LLMs never do?

#158

Post by zer00eyz »

choeger wrote: post #137
zer00eyz wrote: post #133

Earlier quoted context omitted.

Yes: An LLM isnt a model of human thinking. An LLM is an attempt to build a simulation of human communication. An LLM is to language what a forecast is to weather. No amount of weather data is actually going to turn that simulation into snow, no amount of LLM data is going to create AGI. That having been said, better models (smaller, more flexible ones) are going to result in a LOT of practical uses that have the pot…

Great comment. Just one thought: Language, unlike weather, is meta-circular. All we know about specific words or sentences is again encoded in words and sentences. So the embedding encodes a subset of human knowledge. Hence, a LLM is predicting not only language but language with some sort of meaning.

That re-embeding is also encoded in weather. It is why perfect forecasting is impossible, why we talk about the butterfly effect.

The "hallucination problem" is simply the tyranny of Lorenz... one is not sure if a starting state will have a good outcome or swing wildly. Some good weather models are based on re-runing with tweaks to starting params, and then things that end up out of bounds can get tossed. Its harder to know when a result is out of bounds for an LLM, and we dont have the ability to run every request 100 times through various models to get an "average" output yet... However some of the reuse of layers does emulate this to an extent....

Re: What can LLMs never do?

#159

Post by Der_Einzige »

kromem wrote: post #140

LLMs can't is such an anti-pattern at this point I'm surprised that anyone still dares to stake it. The piece even has an example of a $10k bet around a can't being proven false in under a day, but somehow doesn't think maybe their own can't examples are on similarly thin ice? In particular, the line about "what models can't do tells us what they don't know" is infuriating. No, that's not the case at all. At least in…

Amen brother. I had feelings like this which I wanted to share, but you hit the nail on the head.
Post reply on HN