Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Large models of what? Mistaking engineering achievements for linguistic agency

arxiv.org

Post reply on HN
Print view View original

121–130 of 162 posts

Re: Large models of what? Mistaking engineering achievements for linguistic agency

#121

Post by randcraw »

Salgat wrote: post #106
GeneralMayhem wrote: post #8

I am highly skeptical of LLMs as a mechanism to achieve AGI, but I also find this paper fairly unconvincing, bordering on tautological. I feel similarly about this as to what I've read of Chalmers - I agree with pretty much all of the conclusions, but I don't feel like the text would convince me of those conclusions if I disagreed; it's more like it's showing me ways of explaining or illustrating what I already belie…

To me LLMs seem to most closely resemble the regions of the brain used for converting speech to abstract thought and vice-versa, because LLMs are very good at generating natural language and knowing the flow of speech. An LLM is similar to if you took the the Wernicke's and Broca's Areas and stuck a regression between them. The problem is that the regression in the middle is just a brute force of the entire world's k…

I think the major lessons from the success of LLMs are two: 1) the astonishing power of a largely trivial association engine based only on the semantic categories inferred by word2vec, and 2) that so much of the communication abilities of the human mind require so little rational thought (since LLMs demonstrate essentially none of the skills in Kahneman's and Tversky's System 2 thinking (logic, circumspection, self-correction, reflection, etc).

I guess this also disproves Minsky's 'Society of Mind' conjecture - a large part of human cognition (System 1) does not require the complex interaction of heterogeneous mental components.

Re: Large models of what? Mistaking engineering achievements for linguistic agency

#122

Post by nickpsecurity »

jdietrich wrote: post #59
wonnage wrote: post #36

Earlier quoted context omitted.

Or maybe they're just pattern matching on the very particular sentence structure you've chosen. This isn't a convincing example at all

This isn't something I should convince you of. Just open up ChatGPT or Claude and try it for yourself. Think up a batch of your own questions and see how a modern LLM fares. I assure you that it'll do much better than chance. If you're so inclined, you can run enough tests to achieve statistical significance in the course of your lunch break. It depresses me that we seem to be spending more time arguing and hypothesi…

Given their training set, our hypothesis so far should be that they're just tweaking things they've already seen by applying a series of simple rules. They're still not doing what human beings do. We have introspection, creativity operating outside what we've seen, modeling others' thoughts, planning in new domains, and so on. We also operate without hallucination most of the time. I've yet to see an A.I. do all of this reliably and consistently. Then, that it did that without training input similar to the output.

So, they don't just pattern match or purely memorize. They do more than that. They do way less than humans. Unlike humans, they also try to do everything with one or a few components vs our (100-200?) brain components. Crossing that gap might be achievable. It will not be done by current architectures, though.

Re: Large models of what? Mistaking engineering achievements for linguistic agency

#123

Post by [deleted] »

ainoobler wrote: post #119
bubblyworld wrote: post #114

Earlier quoted context omitted.

Well, you know, elaborate and we can have a productive discussion. The way you keep appealing to computability theory as a black box makes me think you haven't actually studied that much of it.

Not much to discuss.

[deleted]

Re: Large models of what? Mistaking engineering achievements for linguistic agency

#124

Post by skybrian »

imtringued wrote: post #69
Lerc wrote: post #28

Earlier quoted context omitted.

I'm not so sure that view is very widespread amongst people familiar with how LLMs work. Certainly they become more capable with parameters and data, but there are fundamental things that can't be overcome with a basic model and I don't think anyone is seriously arguing otherwise. For instance LLMs are pretty much stateless without their context window. If you treat the raw generated output as the first and final res…

The problem with is that you need the internal monologue to not be subject to training loss, otherwise the internal monologue is restricted to the training distribution. Something people don't seem to grasp is that the training data mostly doesn't contain any reasoning. Nobody has published brain activity recordings on the internet, only text written in human language. People see information, process it internally in…

That does sounds invasive, but on the other hand, math teachers do tell the kids to “show their work” for good reasons. And the consent issues don’t apply for LLM training.

I wonder if the trend towards using synthetic, AI-generated training data will make it easier to train models that use effectively? AI’s could be trained to use reasoning and show their work more than people normally do when posting on the Internet. It’s not going to create information out of nothing, but it will better model the distribution that the researchers want the LLM to have, rather than taking distributions found on the Internet as given.

It’s not a natural distribution anyway. For example, I believe it’s already the case that people train AI with weighted distributions - training more on Wikipedia, for example.

My guess is that the quest for the best training data has only just begun.

Re: Large models of what? Mistaking engineering achievements for linguistic agency

#125

Post by exe34 »

zoogeny wrote: post #115
exe34 wrote: post #46

Earlier quoted context omitted.

> . I feel similarly about this as to what I've read of Chalmers - I agree with pretty much all of the conclusions, but I don't feel like the text would convince me of those conclusions if I disagreed; my limited experience of reading Chalmers is that he doesn't actually present evidence - he goes on a meandering rant and then claims to have proved things that he didn't even cover. it was the most infuriating read of…

I haven't read any Chalmers so I can't comment on his writing style. I have seen him in several videos on discussion panels and on podcasts. One thing I appreciate is he often states his premises, or what modern philosophers seem to call "commitments". I wouldn't go so far as to say he uses air-tight logic to reason from these premises/commitments to conclusions - but at the least his reasoning doesn't seem to stray…

yeah while reading the book he would keep saying things that are factually wrong or just state that things are impossible, basically he builds the conclusion into the premises and then discovers the conclusions like he just defended them.

Re: Large models of what? Mistaking engineering achievements for linguistic agency

#126

Post by corimaith »

8n4vidtmkvmk wrote: post #57
sriku wrote: post #44

Earlier quoted context omitted.

The crux of the video game analogy seems to be that when you go close to an object, the resolution starts blurring and the illusion gets broken, and there is a similar thing that happens with LLMs (as of today) as well. This is, so far, reasonable based on daily experience with these models. The extension of that argument being made in the paper is that a model trained on language tokens spewed by humans is incapable…

Why are LLMs incapable of reaching that limit? It's very easy to imagine video games getting to that point. We have all the data to see objects right down to the atomic level, which is plenty more than you'd need for a game. It's mostly a matter of compute. Why then should LLMs breakdown if they can at least mimic the smartest humans? We don't need "resolution" beyond that.

That depends if you believe natural language alone is sufficient to fully model reality. Probably not, it can approximate to a high degree, but there is a reason we resort to formal, constructed languages in math or CS to express our ideas.

Re: Large models of what? Mistaking engineering achievements for linguistic agency

#127

Post by GeneralMayhem »

randcraw wrote: post #121
Salgat wrote: post #106

Earlier quoted context omitted.

To me LLMs seem to most closely resemble the regions of the brain used for converting speech to abstract thought and vice-versa, because LLMs are very good at generating natural language and knowing the flow of speech. An LLM is similar to if you took the the Wernicke's and Broca's Areas and stuck a regression between them. The problem is that the regression in the middle is just a brute force of the entire world's k…

I think the major lessons from the success of LLMs are two: 1) the astonishing power of a largely trivial association engine based only on the semantic categories inferred by word2vec, and 2) that so much of the communication abilities of the human mind require so little rational thought (since LLMs demonstrate essentially none of the skills in Kahneman's and Tversky's System 2 thinking (logic, circumspection, self-c…

What makes this tough is that LLMs can show logical thinking and self-correction when specifically prompted (e.g. "think step by step", "double-check and then correct your work"). It seems unlikely that they can truthfully self-reflect, but I don't think it's strictly impossible.

Re: Large models of what? Mistaking engineering achievements for linguistic agency

#128

Post by TeMPOraL »

zoogeny wrote: post #113
GeneralMayhem wrote: post #8

I am highly skeptical of LLMs as a mechanism to achieve AGI, but I also find this paper fairly unconvincing, bordering on tautological. I feel similarly about this as to what I've read of Chalmers - I agree with pretty much all of the conclusions, but I don't feel like the text would convince me of those conclusions if I disagreed; it's more like it's showing me ways of explaining or illustrating what I already belie…

> On embodiment - yes, LLMs do not have corporeal experience. My own thought on this (as someone who believes embodiment is essential) is to consider the rebuttals to Searle's Chinese Room thought experiment. For now (and the foreseeable future) humans are the embodiment of LLMs. In some sense, we could be seen as playing the role of a centralized AIs nervous system.

Rebuttals of Chinese rooms are also rebuttals of embodiment as a requirement! To say the system of person+books speaks Chinese is to say that good enough emulation of a process has all the qualities of the emulated process, and can substitute for it. Embodiment then cannot be essential, because we could emulate it instead.

Re: Large models of what? Mistaking engineering achievements for linguistic agency

#129

Post by TeMPOraL »

corimaith wrote: post #126
8n4vidtmkvmk wrote: post #57

Earlier quoted context omitted.

Why are LLMs incapable of reaching that limit? It's very easy to imagine video games getting to that point. We have all the data to see objects right down to the atomic level, which is plenty more than you'd need for a game. It's mostly a matter of compute. Why then should LLMs breakdown if they can at least mimic the smartest humans? We don't need "resolution" beyond that.

That depends if you believe natural language alone is sufficient to fully model reality. Probably not, it can approximate to a high degree, but there is a reason we resort to formal, constructed languages in math or CS to express our ideas.

LLMs aren't trained solely on natural language. They also ingest formal notation from every domain and at every level (from preschool to PhD); they see code and markup in every language even remotely popular. They see various encodings, binary dumps, and nowadays also diagrams. The training data has all that's needed to teach them great many formal languages and how to use them.

Re: Large models of what? Mistaking engineering achievements for linguistic agency

#130

Post by [deleted] »

GeneralMayhem wrote: post #127
randcraw wrote: post #121

Earlier quoted context omitted.

I think the major lessons from the success of LLMs are two: 1) the astonishing power of a largely trivial association engine based only on the semantic categories inferred by word2vec, and 2) that so much of the communication abilities of the human mind require so little rational thought (since LLMs demonstrate essentially none of the skills in Kahneman's and Tversky's System 2 thinking (logic, circumspection, self-c…

What makes this tough is that LLMs can show logical thinking and self-correction when specifically prompted (e.g. "think step by step", "double-check and then correct your work"). It seems unlikely that they can truthfully self-reflect, but I don't think it's strictly impossible.

[deleted]
Post reply on HN