The Myth of AI Omniscience: AI's Epistemological Limits • Hacker News Board

Re: The Myth of AI Omniscience: AI's Epistemological Limits

#51

Post by visarga » Sat, Aug 05, 2023, 4:50 PM UTC

Capricorn2481 wrote: post #42

johndough wrote: post #33

> To sum up: because LLMs are fundamentally limited to i) using our vocabulary, ii) “understanding” concepts in the ways we do, and iii) “talking” in the ways we do, then, at best, a LLM can only mirror back to us the order we have carved, the truth we have honed. That conclusion is not correct. Generative models can combine existing concepts in novel ways which have never been considered before. This is most easily…

That's not an LLM, though. We haven't seen evidence yet that an LLM can combine existing language in novel ways. In fact, we've seen over and over that LLMs are quite generic. Compared to text-to-image, which is seemingly impossible to use without getting something weird

> We haven't seen evidence yet that an LLM can combine existing language in novel ways.

How about BubbleSort written in Shakespeare style?

Re: The Myth of AI Omniscience: AI's Epistemological Limits

#52

Post by CuriouslyC » Sat, Aug 05, 2023, 4:51 PM UTC

This is plainly false. With reinforcement learning, an objective function and the ability to generate random solutions it's possible for AI to stumble upon sequences/strategies that greatly outperform human state of the art.

Re: The Myth of AI Omniscience: AI's Epistemological Limits

#53

Post by osmarks » Sat, Aug 05, 2023, 4:52 PM UTC

Capricorn2481 wrote: post #42

johndough wrote: post #33

> To sum up: because LLMs are fundamentally limited to i) using our vocabulary, ii) “understanding” concepts in the ways we do, and iii) “talking” in the ways we do, then, at best, a LLM can only mirror back to us the order we have carved, the truth we have honed. That conclusion is not correct. Generative models can combine existing concepts in novel ways which have never been considered before. This is most easily…

That's not an LLM, though. We haven't seen evidence yet that an LLM can combine existing language in novel ways. In fact, we've seen over and over that LLMs are quite generic. Compared to text-to-image, which is seemingly impossible to use without getting something weird

This is mostly a property of the RLHF/instruction-tuned chatbot models most people use. Base models tend to be more varied.

Re: The Myth of AI Omniscience: AI's Epistemological Limits

#54

Post by visarga » Sat, Aug 05, 2023, 4:54 PM UTC

mrcode007 wrote: post #40

famouswaffles wrote: post #9

LLMs don't learn to simulate or mimic, that's just a byproduct. They learn to predict the training corpus. There is absolutely nothing about the act of prediction that necessitates an upper bound of intelligence on the corpus itself. https://www.pnas.org/doi/full/10.1073/pnas.2016239118 They found representations on fundamental properties of proteins such as secondary structure, contacts, and biological activity in a…

You mentioned the corpus prediction being the core of the LLM. Because of this, my prediction is we will see way more data withholding to prevent LLM learning just like we’ve seen with stack overflow, Reddit, X. I myself have started doing this. For example, I don’t publish code on GitHub anymore to prevent copilot training on my own code. Normally I like to get paid for work, instead of paying for GitHub and doing w…

You are not fundamentally wrong, but there are degrees. LLMs can do some symbolic manipulation and a few reasoning steps. Maybe they are just the result of pattern matching, but I have a hunch that most of the time humans do the same. We fake our understanding as much as we can, we use all shortcuts.

Re: The Myth of AI Omniscience: AI's Epistemological Limits

#55

Post by RoboTeddy » Sat, Aug 05, 2023, 4:55 PM UTC

> But the ways in which a LLM can “talk about” the universe (and everything it contains) are limited to the ways in which humans have previously talked about the universe.

This is often said, but it isn't so.

The task of predicting the next token in human speech really well requires immense intelligence — potentially far more intelligence than possessed by the original speaker! Imagine yourself engaging in the task of listening to someone who isn't that smart speak and then trying to figure out what they'll say next — in doing so, you might make all sorts of extrapolations about the person, their motivations, their manner, their dialect, etc — calling on all sorts of internal models that you've built up about people over time. This is what models are being trained to do when we train them on predicting tokens.

There are concrete examples of models inventing new ways of thinking that are not described in their training set. For example, when training a transformer from scratch to perform addition mod P (and having no training data other than examples of addition mod P), the transformer was able to discover the use of discrete fourier transforms and trigonometric identities [1]. As we can see, neural nets can build all sorts of internal mental models that no one explained to them beforehand. These internal mental models can then be elicited and used for other purposes by e.g. fine-tuning.

I think a good mental model for transformers/neural nets is that they're automatic scientists. They figure out ways of modeling things in order to predict the output from the input — which is what scientists do! As part of this, they can de-facto discover new theories, and come to rely on the theories that prove useful in their prediction task.

Also, not all tokens in the training set are from human speech, so models are being trained to model all manner of data-generating processes.

[1] https://arxiv.org/pdf/2301.05217.pdf

Re: The Myth of AI Omniscience: AI's Epistemological Limits

#56

Post by lkbm » Sat, Aug 05, 2023, 4:58 PM UTC

wrsh07 wrote: post #34

simonh wrote: post #16

The article is claiming that advanced AIs cannot become more intelligent than humans, essentially because LLMs cannot become more intelligent than humans. LLMs aren’t the be all and end all of AI though. They’re an impressive but inherently limited stepping stone, with a very constrained scope of applicability and capabilities. There is no reason to suppose that future, much more advanced architectures can’t surpass…

I think the paper would similarly claim something like "a chess ai trained only on human games can't play better than the humans who played those games" I think that's false - you could imagine it playing at GM level without making mistakes that real GMs make But it's missing an important fact: chess ais aren't limited to learning only from human games. They can learn from self play. You can extend this to physics (i…

It's also not learning from individual humans. It's learning from all the humans.

Different humans have different strengths, including within chess.

Re: The Myth of AI Omniscience: AI's Epistemological Limits

#57

Post by visarga » Sat, Aug 05, 2023, 5:03 PM UTC

zarzavat wrote: post #21

dvt wrote: post #10

Earlier quoted context omitted.

> This is not true and is easy enough to test. How exactly is this not true? Embeddings are literally a mapping of (English) words to numbers.

It’s not a mapping of words. It’s a mapping of character sequences. You can ask chatgpt to define “hackernewsitis (zero hits on google) and it gives a plausible definition. Also the fact that it can write code is evidence that it can understand new concepts. A variable declaration is a coining of a (very short lived) new word.

Hackernewsitis is a made-up term that refers to the tendency of some people on the online forum Hacker News to get overly focused on or obsessed with certain topics, technologies, or companies.

Some characteristics of hackernewsitis include:

- Endlessly debating or commenting on the same subjects over and over, even when no new information is being contributed.

- Getting emotionally invested in certain technologies or companies, treating them like sports teams to cheer for rather than objectively evaluating them.

- Assuming the conversations and interests of Hacker News represent the tech industry as a whole, losing perspective on what is niche vs. mainstream.

- Spending too much time on Hacker News reading and commenting rather than working on your own projects and goals.

- Seeing the world overly in terms of tech industry buzzwords, losing appreciation of other domains.

- Acting overly cynical or pessimistic about new technologies based on theoretical risks rather than evidence.

So in summary, "hackernewsitis" refers to the potential for discussions on Hacker News to become unproductive echochambers if participants are not self-aware. It highlights the need for individuals to think critically and not get swept up in groupthink dynamics.

(Claude)

Me: Does this look like AI not understanding?

Re: The Myth of AI Omniscience: AI's Epistemological Limits

#58

Post by lacrimacida » Sat, Aug 05, 2023, 5:09 PM UTC

bloaf wrote: post #49

Capricorn2481 wrote: post #42

Earlier quoted context omitted.

That's not an LLM, though. We haven't seen evidence yet that an LLM can combine existing language in novel ways. In fact, we've seen over and over that LLMs are quite generic. Compared to text-to-image, which is seemingly impossible to use without getting something weird

There was a post here on hacker news not that long ago where GPT4 came up with "the confetti has left the cannon" when asked for an original phrase similar to "the cat is out of the bag". Other users confirmed Google could not identify any other use of that phrase. People saying GPT4 is unoriginal have an uphill battle, it is not the default assumption of anyone who has worked with it.

ChatGpt is as original as us but cannot ever surpass us because it needs our feedback loop. It is a mirroring of sorts after all.

If it could become extremely creative, push the limits and make ideas so advanced that we can’t understand then it failed at the task so to speak. It always remix idea into larger idea sallads and get our upvotes downvotes. It is a great tool but a tool nonetheless.

Re: The Myth of AI Omniscience: AI's Epistemological Limits

#59

Post by jncfhnb » Sat, Aug 05, 2023, 5:14 PM UTC

Capricorn2481 wrote: post #42

johndough wrote: post #33

> To sum up: because LLMs are fundamentally limited to i) using our vocabulary, ii) “understanding” concepts in the ways we do, and iii) “talking” in the ways we do, then, at best, a LLM can only mirror back to us the order we have carved, the truth we have honed. That conclusion is not correct. Generative models can combine existing concepts in novel ways which have never been considered before. This is most easily…

That's not an LLM, though. We haven't seen evidence yet that an LLM can combine existing language in novel ways. In fact, we've seen over and over that LLMs are quite generic. Compared to text-to-image, which is seemingly impossible to use without getting something weird

That distinction is so pedantic I don’t think it’s worth mentioning. LLM architecture can be extended to other contexts besides language.

Re: The Myth of AI Omniscience: AI's Epistemological Limits

#60

Post by waffletower » Sat, Aug 05, 2023, 5:14 PM UTC

"Therefore, the outputs of a LLM, at best, reflect our current understanding of the universe and nothing more." Others have reflected on the problems with this statement, and I agree, LLMs can be prompted to navigate the training corpus and create novel output, countering this claim. However, these criticisms neglect to credit the external intelligence inherent to the prompts themselves. AutoGPT and its ilk have yet to create convincing agency: for the time being the current state of AI is a mere extension of human intelligence and will not be making lofty discoveries without direct human interaction. LLMs are a form of human co-intelligence.