LLMs and self-referentiality • Hacker News Board

Re: LLMs and self-referentiality

#11

Post by dgellow » Thu, Sep 03, 2026, 7:40 AM UTC

gizajob wrote: post #6

jdw64 wrote: post #4

If self-reference and strange loops aren't intelligence, and emergence is merely a byproduct of universality, then what on earth is intelligence? What is this thing we question as intelligence, and what are the values that take shape from it? Can we truly define intelligence as prediction, prediction as compression, and compression as the process of finding the upper bound of Kolmogorov complexity? Can the statistica…

Confusion around concepts such as intelligence, consciousness, self-awareness, sentience, sapience and aliveness is causing a huge difficulties in the debates around LLMs and AI. Hofstadter didn’t exactly say that self-reference and strange loops were intelligence, but something deeper.

The fact that we constantly switch between the colloquial definitions of those terms and more specific, technical ones doesn’t help

Re: LLMs and self-referentiality

#12

Post by thatjoeoverthr » Thu, Sep 03, 2026, 7:46 AM UTC

Their lack of self reference is a core problem that undergirds a lot of faults that do occur during inference, but their breadth + the agent harness successfully covers it well, so it requires a bit of poking to witness. The “hallucination” phenomenon is exactly this. They don’t know the scope of their own knowledge, and they just say stuff, so if you go out of band, it has a higher probability emitting claims that aren’t true. RAG (I don’t mean embedding indices, but any information ingest such as an agent harness executing a search) are somewhat effective in covering for it, enough to make them very useful! But when it does go wrong, it’s generally the same reasons. It has a certain nature and sometimes you run afoul of it.

But I suppose it doesn’t harm its reasoning!

Re: LLMs and self-referentiality

#13

Post by chrisjj » Thu, Sep 03, 2026, 8:06 AM UTC

An LLM is no more self-referential than a PDF reader showing its user guide.

Re: LLMs and self-referentiality

#14

Post by greyface- » Thu, Sep 03, 2026, 8:15 AM UTC

This one thinks LLMs should be trained and RLHF'd into illeists, as an experiment.

Re: LLMs and self-referentiality

#15

Post by visarga » Thu, Sep 03, 2026, 8:21 AM UTC

I care more about the recursive loop between execution and cost. Anything that executes - be it biological or artificial - incurs some costs. Those costs in turn impact what execution can follow. The gains pay for action, action produces gains.

LLMs are also expensive, not platonic executions. We talk all day long about their costs. Companies developing them are looped with users, investors, competitors and hardware producers. There is a lineage. The self-reference Aaronson can't find in the architecture is in the bill.

Re: LLMs and self-referentiality

#16

Post by someothherguyy » Thu, Sep 03, 2026, 8:23 AM UTC

jdw64 wrote: post #4

If self-reference and strange loops aren't intelligence, and emergence is merely a byproduct of universality, then what on earth is intelligence? What is this thing we question as intelligence, and what are the values that take shape from it? Can we truly define intelligence as prediction, prediction as compression, and compression as the process of finding the upper bound of Kolmogorov complexity? Can the statistica…

there is deeper thought on this topic beyond douglas hofstadter (~20 years ago for strange loop and prior for GEB) and scott aaronson

any cognitive science textbook will bring some thoughts, or a cursory google scholar search

Re: LLMs and self-referentiality

#17

Post by olmo23 » Thu, Sep 03, 2026, 8:23 AM UTC

lmf4lol wrote: post #10

jdw64 wrote: post #4

If self-reference and strange loops aren't intelligence, and emergence is merely a byproduct of universality, then what on earth is intelligence? What is this thing we question as intelligence, and what are the values that take shape from it? Can we truly define intelligence as prediction, prediction as compression, and compression as the process of finding the upper bound of Kolmogorov complexity? Can the statistica…

I concurrently prompt a senior human programmers and a set of AI programmers and the difference is the following: The human applies critical thinking, questions choices, suggests his own ideas and has a certain taste. I also can prompt him, leave for 2 weeks, come back and have a result. And I can trust that it works! Velocity is OK. I often wish he would be faster :-) The AI needs baby sitting and steering. It codes…

I would be extremely offended if I heard someone claim they "prompt" me. It's very disrespectful.

Re: LLMs and self-referentiality

#18

Post by dsign » Thu, Sep 03, 2026, 8:40 AM UTC

> What can we say about this with hindsight? > While the ideas of diagonalization and self-reference of course played a central role in the birth of modern mathematical logic and computer science, the most famous uses were negative.

The two sentences above are in the article, but I'm taking them out of context because they are my take on this entire AI brouhaha. There are a lot of "negatives" in our cultural reference frames. One of those, extremely pervasive, is that "humans were made by God". I could write that statement as "humans are exceptional in a way that can't be replicated", which might be ideologically softer, but then I would be taking a long roundabout to make my point. Which is that, after praying and worshiping for thousands of years, and (yes! yes!) sculpting our language and our sagas to account for and praise the divine and its intent, then there's little mystery in our many, many attempts to reify in mathematics and logic our purely cultural framing. It doesn't matter how much of an atheist a thinker is, they still have tons of transitive faith.

Re: LLMs and self-referentiality

#19

Post by aswegs8 » Thu, Sep 03, 2026, 8:47 AM UTC

Yeah but that is the whole point, right? Maybe I am thinking too simply, but it seems obvious to me. LLMs are intelligence without consciousness. Once we build in a self-referential learning loop into the models, it will gain a form of what we consider consciousness.

Re: LLMs and self-referentiality

#20

Post by jwr » Thu, Sep 03, 2026, 8:51 AM UTC

I think before we get to self-referentiality, we have a bigger problem to solve, and that's one that I did not expect: namely, at which point do we stop saying that a machine pretends to be conscious and start saying that it is conscious?

We have, somewhat unexpectedly, built machines that are very, very good at pretending. Now, I'm not calling the current generation of LLMs we have "conscious," but I can't really define a marker or a boundary beyond which I would start calling them "conscious."