What can LLMs never do?

Post by spacebanana7 » Sat, Apr 27, 2024, 6:58 PM UTC

mewpmewp2 wrote: post #91

Mordisquitos wrote: post #45

Earlier quoted context omitted.

I think that the article is correct. There are indeed things that L L Ms will never be able to do, at least not consistently, however much the hardware improves or on how much more material they are trained. How come? Note my emphasis on the 2nd 'L'. I'm not saying that there are things that AI models will never be able to do, I'm saying that there are things that Large Language Models will be unable to do. Training…

I don't think values, opinions or things like that are needed at all. These are just aspects we have in order to perform in and together with the society. Also doubt is just uncertainty, and can be represented as a probability. Actually all values and everything can be presented as a numerical probability, which I personally prefer to do as well.

Post by martindbp » Sat, Apr 27, 2024, 7:42 PM UTC

jillesvangurp wrote: post #87

shawntan wrote: post #71

If we're trying to quantify what they can NEVER do, I think we'd have to resort to some theoretical results rather than a list empirical evidence of what they can't do now. The terminology you'd look for in the literature would be "expressibility". For a review of this topic, I'd suggest: https://nessie.ilab.sztaki.hu/~kornai/2023/Hopf/Resources/st... The authors of this review have themselves written several article…

We have to be a bit more honest about the things we can actually do ourselves. Most people I know would flunk most of the benchmarks we use to evaluate LLMs. Not just a little bit but more like completely and utterly and embarrassingly so. It's not even close; or fair. People are surprisingly alright at a narrow set of problems. Particularly when it doesn't involve knowledge. Most people also suck at reasoning (unles…

Post by wave_1 » Sat, Apr 27, 2024, 8:11 PM UTC

Post by asicsarecool » Sat, Apr 27, 2024, 8:39 PM UTC

wave_1 wrote: post #105

I build an Agentic AI that leverages #6 and #7 at the end of the article as well as techniques not yet published. It tackles hallucination relative not to the world at large but to the facts, entities and causal relationships contained in a document (which is really bad reasoning if we assume LLMs are "reasoning" to begin with) It also tackles cross-reasoning with very large token distance. https://www.youtube.com/wa…

Post by esafak » Sat, Apr 27, 2024, 8:46 PM UTC

cs702 wrote: post #60

cs702 wrote: post #14

Fantastic essay. Highly recommended! I agree with all key points: * There are problems that are easy for human beings but hard for current LLMs (and maybe impossible for them; no one knows). Examples include playing Wordle and predicting cellular automata (including Turing-complete ones like Rule 110). We don't fully understand why current LLMs are bad at these tasks. * Providing an LLM with examples and step-by-step…

As an aside, at one point I experimented a little with transformers that had access to external memory searchable via KNN lookups https://github.com/lucidrains/memorizing-transformers-pytorc... ( great work by lucidrains) or via routed queries with https://github.com/glassroom/heinsen_routing (don't fully understand it; apparently related to attention). Both approaches seemed to work, but I had to put that work on ho…