Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The Myth of AI Omniscience: AI's Epistemological Limits

cpwalker.substack.com

Post reply on HN
Print view View original

41–50 of 107 posts

Re: The Myth of AI Omniscience: AI's Epistemological Limits

#41

Post by lsy »

famouswaffles wrote: post #9

LLMs don't learn to simulate or mimic, that's just a byproduct. They learn to predict the training corpus. There is absolutely nothing about the act of prediction that necessitates an upper bound of intelligence on the corpus itself. https://www.pnas.org/doi/full/10.1073/pnas.2016239118 They found representations on fundamental properties of proteins such as secondary structure, contacts, and biological activity in a…

The problem is that new knowledge comes not only from combination but discovery, and discovery fundamentally requires access to and interaction with the actual world. And as of yet we have not created a way that computers can directly access the world, all I/O is mediated, etc. The more pedantic argument would be that what an LLM can write about the world is fundamentally limited to what has thus far been captured in human writing, and any structural properties thereof.

To your point, prediction may not be bound by what the data explicitly shows, but it is necessarily bound by the data and its implicit and explicit structure. But to argue that the LLM can make any discovery a person can, or interact fluidly with the world as it exists, would require one to believe that every property of the world is either explicitly captured in human-written text or in the structure of that text, which I think would be pretty difficult to argue.

Re: The Myth of AI Omniscience: AI's Epistemological Limits

#42

Post by Capricorn2481 »

johndough wrote: post #33

> To sum up: because LLMs are fundamentally limited to i) using our vocabulary, ii) “understanding” concepts in the ways we do, and iii) “talking” in the ways we do, then, at best, a LLM can only mirror back to us the order we have carved, the truth we have honed. That conclusion is not correct. Generative models can combine existing concepts in novel ways which have never been considered before. This is most easily…

That's not an LLM, though. We haven't seen evidence yet that an LLM can combine existing language in novel ways. In fact, we've seen over and over that LLMs are quite generic.

Compared to text-to-image, which is seemingly impossible to use without getting something weird

Re: The Myth of AI Omniscience: AI's Epistemological Limits

#43

Post by visarga »

> Future LLMs, regardless of model architecture, are fundamentally constrained in this way, by virtue of the fact that they are trained on human-written texts. Therefore, the outputs of a LLM, at best, reflect our current understanding of the universe and nothing more.

That constraint holds only for pure LLMs. There is a secondary source of learning - external feedback to the model. When a LLM is part of a larger system, it can integrate feedback to improve itself, like AlphaGo Zero. A LLM+compiler could self-optimize by observing compilation errors and program outputs. A LLM+game could self-improve by conditioning actions on the score. Humans could create a preference dataset and model for RLHF to refine the LLM based on human feedback. A LLM+robot could learn from past experiences to improve its planning.

Similarly, humans need tools to conduct research. We can't do pure research without labs and experiments. LLMs need external confirmation the same way we do. The outputs of a pure LLM merely reflect our current understanding of the universe. But coupled with real-world feedback, LLMs have the potential to learn and discover new knowledge.

Re: The Myth of AI Omniscience: AI's Epistemological Limits

#44

Post by glenstein »

Capricorn2481 wrote: post #42
johndough wrote: post #33

> To sum up: because LLMs are fundamentally limited to i) using our vocabulary, ii) “understanding” concepts in the ways we do, and iii) “talking” in the ways we do, then, at best, a LLM can only mirror back to us the order we have carved, the truth we have honed. That conclusion is not correct. Generative models can combine existing concepts in novel ways which have never been considered before. This is most easily…

That's not an LLM, though. We haven't seen evidence yet that an LLM can combine existing language in novel ways. In fact, we've seen over and over that LLMs are quite generic. Compared to text-to-image, which is seemingly impossible to use without getting something weird

Is there something about the nature of language and linguistic meaning that would make it difficult to combine language in novel ways?

Re: The Myth of AI Omniscience: AI's Epistemological Limits

#45

Post by masswerk »

famouswaffles wrote: post #9

LLMs don't learn to simulate or mimic, that's just a byproduct. They learn to predict the training corpus. There is absolutely nothing about the act of prediction that necessitates an upper bound of intelligence on the corpus itself. https://www.pnas.org/doi/full/10.1073/pnas.2016239118 They found representations on fundamental properties of proteins such as secondary structure, contacts, and biological activity in a…

However, any such information must be already contained in the semantic field that is provided to the LLM. Either explicitly, or as "nascent" relations in the data that may align with the structure of concepts in that data, since what we may regard as concepts in the LLM is either a matter of embedding or a matter of gradient descent (and related combinatory productions) – and thus again limited by the structure of the semantic field.

Re: The Myth of AI Omniscience: AI's Epistemological Limits

#46

Post by Davidzheng »

lsy wrote: post #41
famouswaffles wrote: post #9

LLMs don't learn to simulate or mimic, that's just a byproduct. They learn to predict the training corpus. There is absolutely nothing about the act of prediction that necessitates an upper bound of intelligence on the corpus itself. https://www.pnas.org/doi/full/10.1073/pnas.2016239118 They found representations on fundamental properties of proteins such as secondary structure, contacts, and biological activity in a…

The problem is that new knowledge comes not only from combination but discovery , and discovery fundamentally requires access to and interaction with the actual world. And as of yet we have not created a way that computers can directly access the world, all I/O is mediated, etc. The more pedantic argument would be that what an LLM can write about the world is fundamentally limited to what has thus far been captured i…

I think discovery for llm is possible both through its knowledge of the world from training data and also from some fields which are theoretical enough that interaction is not needed for discover--e.g pure mathematics but also many applied science

Re: The Myth of AI Omniscience: AI's Epistemological Limits

#47

Post by [deleted] »

jstanley wrote: post #28
dvt wrote: post #26

Earlier quoted context omitted.

> It’s not a mapping of words. It’s a mapping of character sequences. This is plainly wrong if you read the word2vec paper.

It depends which tokenisation you use. Here's the token list for GPT-4: https://incoherency.co.uk/interest/gpt4-token-list.txt Some of them are words, some of them are not. It uses the same list even if you have read the word2vec paper. It probably doesn't even know whether you've read it.

[deleted]

Re: The Myth of AI Omniscience: AI's Epistemological Limits

#48

Post by doug_durham »

Capricorn2481 wrote: post #42
johndough wrote: post #33

> To sum up: because LLMs are fundamentally limited to i) using our vocabulary, ii) “understanding” concepts in the ways we do, and iii) “talking” in the ways we do, then, at best, a LLM can only mirror back to us the order we have carved, the truth we have honed. That conclusion is not correct. Generative models can combine existing concepts in novel ways which have never been considered before. This is most easily…

That's not an LLM, though. We haven't seen evidence yet that an LLM can combine existing language in novel ways. In fact, we've seen over and over that LLMs are quite generic. Compared to text-to-image, which is seemingly impossible to use without getting something weird

I have absolutely seen an LLM combine language in novel ways. With prompting they will create outputs that are entirely novel with concepts intermingled in new ways. It's nonsense when compared to observed reality, but it is novel nonetheless.

Re: The Myth of AI Omniscience: AI's Epistemological Limits

#49

Post by bloaf »

Capricorn2481 wrote: post #42
johndough wrote: post #33

> To sum up: because LLMs are fundamentally limited to i) using our vocabulary, ii) “understanding” concepts in the ways we do, and iii) “talking” in the ways we do, then, at best, a LLM can only mirror back to us the order we have carved, the truth we have honed. That conclusion is not correct. Generative models can combine existing concepts in novel ways which have never been considered before. This is most easily…

That's not an LLM, though. We haven't seen evidence yet that an LLM can combine existing language in novel ways. In fact, we've seen over and over that LLMs are quite generic. Compared to text-to-image, which is seemingly impossible to use without getting something weird

There was a post here on hacker news not that long ago where GPT4 came up with "the confetti has left the cannon" when asked for an original phrase similar to "the cat is out of the bag".

Other users confirmed Google could not identify any other use of that phrase.

People saying GPT4 is unoriginal have an uphill battle, it is not the default assumption of anyone who has worked with it.

Re: The Myth of AI Omniscience: AI's Epistemological Limits

#50

Post by osmarks »

lsy wrote: post #41
famouswaffles wrote: post #9

LLMs don't learn to simulate or mimic, that's just a byproduct. They learn to predict the training corpus. There is absolutely nothing about the act of prediction that necessitates an upper bound of intelligence on the corpus itself. https://www.pnas.org/doi/full/10.1073/pnas.2016239118 They found representations on fundamental properties of proteins such as secondary structure, contacts, and biological activity in a…

The problem is that new knowledge comes not only from combination but discovery , and discovery fundamentally requires access to and interaction with the actual world. And as of yet we have not created a way that computers can directly access the world, all I/O is mediated, etc. The more pedantic argument would be that what an LLM can write about the world is fundamentally limited to what has thus far been captured i…

"Directly access"? Computers can use cameras and robotics hardware and such. It is not as if humans are special because of using eyes and muscles instead.

Also, text does say a lot about the world. It includes descriptions of physics experiments and people's best knowledge of the laws, for instance.

Post reply on HN