Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

AGI is Mathematically Impossible 2: When Entropy Returns

philarchive.org

Post reply on HN
Print view View original

71–80 of 437 posts

Re: AGI is Mathematically Impossible 2: When Entropy Returns

#71

Post by ICBTheory »

Animats wrote: post #42

Penrose did this argument better.[1] Penrose has been making that argument for thirty years, and it played better before AI started getting good. AI via LLMs has limitations, but they don't come from computability. [1] https://sortingsearching.com/2021/07/18/roger-penrose-ai-ske...

Thanks — and yes, Penrose’s argument is well known.

But this isn’t that, as I’m not making a claim about consciousness or invoking quantum physics or microtubules (which, I agree, are highly speculative).

The core of my argument is based on computability and information theory — not biology. Specifically: that algorithmic systems hit hard formal limits in decision contexts with irreducible complexity or semantic divergence, and those limits are provable using existing mathematical tools (Shannon, Rice, etc.).

So in some way, this is the non-microtubule version of AI critique. I don’t have the physics background to engage in Nobel-level quantum speculation — and, luckily, it’s not needed here.

Re: AGI is Mathematically Impossible 2: When Entropy Returns

#72

Post by yodon »

ICBTheory wrote: post #2

This paper presents a theoretical proof that AGI systems will structurally collapse under certain semantic conditions — not due to lack of compute, but because of how entropy behaves in heavy-tailed decision spaces. The idea is called IOpenER: Information Opens, Entropy Rises. It builds on Shannon’s information theory to show that in specific problem classes (those with α ≤ 1), adding information doesn’t reduce uncer…

I'm wondering if you may have rediscovered the concept of "Wicked Problems", which have been studied in system analysis and sociology since the 1970's (I'd cite the Wikipedia page, but I've never been particularly fond of Wikipedia's write up on them). They may be worth reading up on if you're not familiar with them.

Re: AGI is Mathematically Impossible 2: When Entropy Returns

#73

Post by kelseyfrog »

> And - as wonderfully remarkable as such a system might be - it would, for our investigation, be neither appropriate nor fair to overburden AGI by an operational definition whose implicit metaphysics and its latent ontological worldviews lead to the epistemology of what we might call a “total isomorphic a priori” that produces an algorithmic world-formula that is identical with the world itself (which would then make the world an ontological algorithm...?).

> Anyway, this is not part of the questions this paper seeks to answer. Neither will we wonder in what way it could make sense to measure the strength of a model by its ability to find its relative position to the object it models. Instead, we chose to stay ignorant - or agnostic? - and take this fallible system called "human". As a point of reference.

Cowards.

That's the main counter argument and acknowledging its existence without addressing it is a craven dodge.

Assuming the assumptions[1] are true, then human intelligence isn't even able to be formalized under the same pretext.

Either human intelligence isn't

1. Algorithmic. The main point of contention. If humans aren't algorithmically reducible - even at the level computation of physics, then human cognition is supernatural.

2. Autonomous. Trivially true given that humans are the baseline.

3. Comprehensive (general): Trivially true since humans are the baseline.

4. Competent: Trivially true given humans are the baseline.

I'm not sure how they reconcile this given that they simply dodge the consequences that it implies.

Overall, not a great paper. It's much more likely that their formalism is wrong than their conclusion.

Footnotes

1. not even the consequences, unfortunately for the authors.

Re: AGI is Mathematically Impossible 2: When Entropy Returns

#74

Post by Takashoo »

proc0 wrote: post #53

The paper is skipping over the definition of AI. It jumps right into AGI, and that depends on what AI means. It could be LLMs, deep neural networks, or any possible implementation on a Turing machine. The latter I suspect would be extremely difficult to prove. So far almost everything can be simulated by Turing machines and there's no reason it couldn't also simulate human brains, and therefore AGI. Even if the claim…

Turing machines only model computation. Real life is interaction. Check the work of Peter Wegner. When interaction machines enter into the picture, AI can be embodied, situated and participate in adaptation processes. The emergent behaviour may bring AGI in a pragmatic perspective. But interaction is far more expressive than computation rendering theoretical analysis challenging.

Re: AGI is Mathematically Impossible 2: When Entropy Returns

#75

Post by like_any_other »

ICBTheory wrote: post #51
like_any_other wrote: post #3

So does the human brain transcend math, or are humans not generally intelligent?

Hi and thanks for engaging :-) Well, it in fact depends on what intelligence is to your understanding: -If it intelligence = IQ, i.e. the rational ability to infer, to detect/recognize and extrapolate patterns etc, then AI is or will soon be more intelligent than us, while we humans are just muddling through or simply lucky having found relativity theory and other innovations just at the convenient moment in time ...…

Let me steal another users alternate phrasing: Since humans and computers are both bound by the same physical laws, why does your proof not apply to humans?

Re: AGI is Mathematically Impossible 2: When Entropy Returns

#76

Post by ICBTheory »

WhitneyLand wrote: post #48
ICBTheory wrote: post #2

This paper presents a theoretical proof that AGI systems will structurally collapse under certain semantic conditions — not due to lack of compute, but because of how entropy behaves in heavy-tailed decision spaces. The idea is called IOpenER: Information Opens, Entropy Rises. It builds on Shannon’s information theory to show that in specific problem classes (those with α ≤ 1), adding information doesn’t reduce uncer…

“This paper presents a theoretical proof that AGI systems will structurally collapse under certain semantic conditions…” No it doesn’t. Shannon entropy measures statistical uncertainty in data. It says nothing about whether an agent can invent new conceptual frames. Equating “frame changes” with rising entropy is a metaphor, not a theorem, so it doesn’t even make sense as a mathematical proof. This is philosophical m…

Correct: Shannon entropy originally measures statistical uncertainty over a fixed symbol space. When the system is fed additional information/data, then entropy goes down, uncertainty falls. This is always true in situations where the possible outcomes are a) sufficiently limited and b)unequally distributed. In such cases, with enough input, the system can collapse the uncertainty function within a finite number of steps.

But the paper doesn’t just restate Shannon.

It extends this very formalism to semantic spaces where the symbol set itself becomes unstable. These situations arise when (a) entropy is calculated across interpretive layers (as in LLMs), and (b) the probability distribution follows a heavy-tailed regime (α ≤ 1). Under these conditions, entropy divergence becomes mathematically provable.

This is far from being metaphorical: it’s backed by formal Coq-style proofs (see Appendix C in he paper).

AND: it is exactly the mechanism that can explain the Apple-Papers' results

Re: AGI is Mathematically Impossible 2: When Entropy Returns

#77

Post by ninetyninenine »

hnfong wrote: post #25
ninetyninenine wrote: post #15

Earlier quoted context omitted.

We don’t even know how LLMs work. But we do know the underlying mechanisms are governed by math because we have a theory of reality that governs things down to the atomic scale and humans and LLMs are made out of atoms. So because of this we know reality is governed by maths. We just can’t fully model the high level consequence of emergent patterns due to the sheer complexity of trillions of interacting atoms. So it’…

> we have a theory of reality that governs things down to the atomic scale and humans and LLMs are made out of atoms. > So because of this we know reality is governed by maths. That's not really true. You have a theory, and let's presume so far it's consistent with observations. But it doesn't mean it's 100% correct, and doesn't mean at some point in the future you won't observe something that invalidates the theory.…

>That's not really true. You have a theory, and let's presume so far it's consistent with observations. But it doesn't mean it's 100% correct, and doesn't mean at some point in the future you won't observe something that invalidates the theory. In short, you don't know whether the theory is absolutely true and you can never know.

Let me repharse it. As far as we know all of reality is governed by the principles of logic and therefore math. This is the most likely possibility and we have based all of our technology and culture and science around this. It is the fundamental assumption humanity has made on reality. We cannot consistently demonstrate disproof against this assumption.

>Not the GP but I think humility is kinda relevant here.

How so? If I assume all of reality is governed by math, but you don't. How does that make me not humble but you humble? Seems personal.

Re: AGI is Mathematically Impossible 2: When Entropy Returns

#78

Post by ninetyninenine »

bigyabai wrote: post #18
ninetyninenine wrote: post #15

Earlier quoted context omitted.

We don’t even know how LLMs work. But we do know the underlying mechanisms are governed by math because we have a theory of reality that governs things down to the atomic scale and humans and LLMs are made out of atoms. So because of this we know reality is governed by maths. We just can’t fully model the high level consequence of emergent patterns due to the sheer complexity of trillions of interacting atoms. So it’…

> We don’t even know how LLMs work Speak for yourself. LLMs are a feedforward algorithm inferring static weights to create a tokenized response string. We can compare that pretty trivially to the dynamic relationship of neurons and synapses in the human brain. It's not similar, case closed. That's the extent of serious discussion that can be had comparing LLMs to human thought, with apologies to Chomsky et. al. It's…

George Hinton the person largely responsible about the AI revolution has this to say:

https://www.reddit.com/r/singularity/comments/1lbbg0x/geoffr...

https://youtu.be/qrvK_KuIeJk?t=284

In that video above George Hinton, directly says we don't understand how it works.

So I don't speak just for myself. I speak for the person who ushered in the AI revolution, I speak for Experts in the field who know what they're talking aboutt. I don't speak for people who don't know what they're talking about.

Even though we know it's a feedforward network and we know how the queries are tokenized you cannot tell me what an LLM would say nor tell me why an LLM said something for a given prompt showing that we can't fully control an LLM because we don't fully understand it.

Don't try to just argue with me. Argue with the experts. Argue with the people who know more than you, Hinton.

Re: AGI is Mathematically Impossible 2: When Entropy Returns

#79

Post by DougN7 »

mort96 wrote: post #43
alganet wrote: post #16

Earlier quoted context omitted.

What makes you think that human intelligence is based on mathematics?

I will answer under the metaphysical assumption that there is no immaterial "soul", and that the entirety of the human experience arises from material things governed by the laws of physics. If you disagree with this assumption, there is no conversation to be had. The laws of physics can, as far as I can tell, be described using mathematics. That doesn't mean that we have a perfect mathematical model of the laws of p…

I wonder if we could ever compute which exact atom in nuclear fission will split at a very specific time. If that is impossible, then our math and understanding of physics is so far short of what is needed that I don’t feel comfortable with your starting assumption.

Re: AGI is Mathematically Impossible 2: When Entropy Returns

#80

Post by ben_w »

geoka9 wrote: post #57
like_any_other wrote: post #3

So does the human brain transcend math, or are humans not generally intelligent?

Humans are fallible in a way computers are not. One could argue any creative process is an exercise in fallibility. More interestingly, humans are capable of assessing the results of their "neural misfires" ("hmm, there's something to this"), whereas even if we could make a computer do such mistakes, it wouldn't know its Penny Lane from its Daddy's Car[0], even if it managed to come up with one. [0] https://www.youtu…

Hang on, hasn't everyone spent the past few years complaining about LLMs and diffusion models being very fallible?

And we can get LLMs to do better by just prompting them to "think step by step" or replacing the first ten attempts to output a "stop" symbolic token with the token for "Wait… "?

Post reply on HN