Live data from Hacker News

The Limits of LLM

news.ycombinator.com

1–6 of 6 posts

The Limits of LLM

#1
It is possible that LLMs use a similar concept than the Jennifer Aniston neuron (we have found that some neurons only fire when we see a particular person's face) https://www.nature.com/articles/news050620-7.

A LLM could have the Donald Trump neuron. Whenever the prompt mentions Trump, the neuron activates and that neuron cascades activation of paricular sentiment neurons and those neurons re-shape all word's probabilities so that at the end of the process, we have text that contains "believe me" and "tremendous" and "a lot of money".

Then it becomes easy for the model to transform any piece of text in the style of Eminem or Shakira or anything for which there is a neuron that represents that concept.

This also goes in hand with the improvement in NLP we see with larger and larger models. Bigger models have more space to assign specific concepts to specific neurons. And bigger models MUST be trained with bigger datasets because if there are no more concepts to extract from the text, the representation of the world that the model learns caps. Like books and bookshelves.

Thinking in those terms hints at the limitations of LLMs because models can't learn things that are not in the books. Unless we are willing to believe that the whole corpus of books out there contain more information than the information intended to be put by the writers. Maybe there is information in the spaces between books that can be harvest by LLM which humans have not yet tapped into.

I think LLMs will not beat the top humans in any particular area of expertise but they will beat everyone else. They will be below top knowledge but way above average.

Re: The Limits of LLM

#2
What LLMs learn is a "conditional probability distribution" that is the probability that a particular word is used contingent on the other words used in the text.

If it learns to write in the style of Donald Trump that's because it sees "in the style of Donald Trump" in the prompt and that elicits it consider what word Donald Trump would use in a situation. If it is asked to "talk like a pirate", it similarly is drawing from a different probability distribution.

It doesn't literally memorize the content of all the books it was fed because it does not have enough neural connections to encode it all without loss. What's miraculous about it is that it models these "conditional probability distributions" very effectively, it doesn't need to see an example of seeing absolutely everything written in "talk like a pirate style", it can somehow blend what it knows about the conditional probabilities of pirate talk and the conditional probabilities of other conversation and frequently produce a convincing result.

So it is very possible it can follow patterns and make inferences based on what it "learned" in pre-training and what it has just been told and come to conclusions that weren't already fed into it.

On the other hand if I want to talk with it about football it is not at all qualified to have an opinion about "Do you think Arsenal or Man City will win the championship this year?" because the LLM doesn't contain any knowledge about this season at all. (Note if you ask Bing Chat it does a search lookup and interprets the results.)

Also note it is all statistical so it is likely to "make stuff up" whether it is facts that it could have read in a book or when you ask it to make some conclusion based on the knowledge available to it. It can very well make up something that wasn't fed it into it originally, whether or not it is right is a roll the dice sort of thing.

Re: The Limits of LLM

#3

What LLMs learn is a "conditional probability distribution" that is the probability that a particular word is used contingent on the other words used in the text. If it learns to write in the style of Donald Trump that's because it sees "in the style of Donald Trump" in the prompt and that elicits it consider what word Donald Trump would use in a situation. If it is asked to "talk like a pirate", it similarly is draw…

from chatgpt

[ME] which famous person would use these words: - a lot of money - tremendous - believe me

[CHATGPT] One famous person who often uses these words is former United States President Donald Trump. He is known for his frequent use of the phrase "believe me" in his speeches and interviews, as well as his use of the word "tremendous" to describe things that he considers to be large or impressive. Trump is also known for his wealth and for frequently mentioning his own wealth or business acumen, making the phrase "a lot of money" also fitting for him.

I am tempted to download one of the available models and start to look for these neurons. I will probably waste my time, but fun is warrantied

Re: The Limits of LLM

#4
post #3

What LLMs learn is a "conditional probability distribution" that is the probability that a particular word is used contingent on the other words used in the text. If it learns to write in the style of Donald Trump that's because it sees "in the style of Donald Trump" in the prompt and that elicits it consider what word Donald Trump would use in a situation. If it is asked to "talk like a pirate", it similarly is draw…

from chatgpt [ME] which famous person would use these words: - a lot of money - tremendous - believe me [CHATGPT] One famous person who often uses these words is former United States President Donald Trump. He is known for his frequent use of the phrase "believe me" in his speeches and interviews, as well as his use of the word "tremendous" to describe things that he considers to be large or impressive. Trump is also…

actually, I think you are right. At least chatgpt agrees with your argument

[CHATGPT] Specifically, my understanding of Donald Trump is not based on a single neuron, but rather on the collective activation of a large number of neurons distributed across the different layers of my neural network. Each neuron in my neural network represents a specific feature or concept that has been learned from the training data, and the activation of these neurons collectively contribute to my understanding of complex concepts like "Donald Trump".

Re: The Limits of LLM

#5
even if what you said is right, you're assuming that LLMs is static and won't make any progress or breakthrough. With the amount of time , energy and money put into LLMs, we should expect more progress in a few years or even months.