Saving for later, thanks!
What can LLMs never do?
121–130 of 385 posts
Re: What can LLMs never do?
#122Why can't we resolve this with synthetic data? Just take the original statements and ask another LLM to flip around the relation, then train on both. I tried this with GPT-4 and it seems to understand the task [0]:
Me:
For each of the following relational statements, flip the relation around and generate a new statement or question answer pair.
Input: Bluey's father is Bandit
Output: Bandit's daughter is Bluey
Input: Neil Armstrong was the first person to step on the Moon.
Output: Who was the first person to step on the Moon? Neil Armstrong.
Now for the real statements:
- Valentina Tereshkova was the first woman to travel to space
- Who is Mary Lee Pfeiffer's son? Tom Cruise
ChatGPT:
Here are the flipped relational statements:
Valentina Tereshkova was the first woman to travel to space
Output: Who was the first woman to travel to space? Valentina Tereshkova.
Who is Mary Lee Pfeiffer's son? Tom Cruise
Output: Tom Cruise's mother is Mary Lee Pfeiffer.
[0]: https://chat.openai.com/share/33eb3ee4-2094-4748-b01e-0967af...
Re: What can LLMs never do?
#123"The models, in other words, do not well generalise to understand the relationships between people."
Curiously, the need to do this well - handling the quadratic complexity of a shifting set of human relationships, grudges, and alliances - is thought to be one of the things that led is to higher levels of intelligence.
Re: What can LLMs never do?
#124Earlier quoted context omitted.
Neural nets can approximate any function. A large enough llm with memory is turning complete. So theoretically I don’t think there is anything they can never do.
> Neural nets can approximate any function. Common misunderstanding of the universal approximation theorem. Consider this: can an mlp approximate a sine wave? > A large enough llm with memory is turning complete. With (a lot of) chain of thought it could be. Read the paper, and its references.
"Consider this: can an mlp approximate a sine wave?"
Well, yes - we have neutral speech and music synthesis and compression algorithms which do this exceedingly well...
Re: What can LLMs never do?
#125Fantastic essay. Highly recommended! I agree with all key points: * There are problems that are easy for human beings but hard for current LLMs (and maybe impossible for them; no one knows). Examples include playing Wordle and predicting cellular automata (including Turing-complete ones like Rule 110). We don't fully understand why current LLMs are bad at these tasks. * Providing an LLM with examples and step-by-step…
>We don't fully understand why current LLMs are bad at these tasks. In complete seriousness, can anyone can explain why LLMs are good at some tasks?
> why LLMs are good at some tasks?
Like how we explain human doing tasks -- they are evolved to do that.
I believe this is a non-answer, but if we are satisfied with that non answer for human, why not LLMs?
Re: What can LLMs never do?
#126Earlier quoted context omitted.
[flagged]
I've read this three times and it still doesn't make a lick of sense. How does this relate to the parent comments?
Your English is absolutely fine and your answers in this thread clearly addressed the points brought up by other commenters. I have no idea what that guy is on about.
Re: What can LLMs never do?
#127I have been trying to generate some text recently using the ChatGPT API. No matter how I word “Include any interesting facts or anecdotes without commenting on the fact being interesting” it ALWAYS starts out “One interesting fact about” or similar phrasing. I have honestly spent multiple hours trying to word the prompt so it will stop including introductory phrases and just include the fact straight. I have gone so…
I just tried this locally with llama3-8b and it handled it fine. Claude 3 Sonnet passes your test too, in case you don't have the hardware for a local model. You might want to consider moving on from ChatGPT since their models have been RLHF'd to hell in the name of "safety" and are falling behind in many ways as a result. --- transcript: $ ollama run llama3:8b >>> tell me an interesting fact about etymology Here's o…
Why is any of this "interesting"?
Is the response interesting because you don't know it? —How does it know this?
Is the response interesting because you do know it or might have offered it yourself? —Confirmation bias.
Is it interesting because a lot of training references contextualize it as "interesting?" —Begged question.
Is it contextually interesting? —What is the context? A robot refers to robots? How unexpected...
Is it interesting within the narrow confines of LLM adaptations to a scope of inputs?
Can their by any more damning claim of the general suitability of the technology as an oracle than different users using the same prompts and getting inexplicably contrary results?
If trivial prompt alignments result in appropriate vs inappropriate responses, this destroys confidence for every response.
What am I missing?
Re: What can LLMs never do?
#128Fantastic essay. Highly recommended! I agree with all key points: * There are problems that are easy for human beings but hard for current LLMs (and maybe impossible for them; no one knows). Examples include playing Wordle and predicting cellular automata (including Turing-complete ones like Rule 110). We don't fully understand why current LLMs are bad at these tasks. * Providing an LLM with examples and step-by-step…
I would argue that the G in AGI means it can't require better prompting.
Re: What can LLMs never do?
#129All points described are simple artifacts of tokenization.
Re: What can LLMs never do?
#130Earlier quoted context omitted.
> If the text that the text-generating tool generates contains reasoning, then the text generation tool can be said to be reasoning, can't it. I don't know... you're still describing a talking parrot here, if you'd ask me.
I’m not a fan of the talking parrot argument, especially when you’re pointing it at models of scale. The only thing separating a talking parrot and humans is our accuracy in shaping our words to the context in which they’re spoken. Sure it’s easy to liken a low resource model to a talking parrot, the output seems no better than selective repetition of training data. But is that really so different from a baby whose f…
What if we were talking with each other and the right answer for me would be to kiss you on the cheek? Then what?