Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

AGI is Mathematically Impossible 2: When Entropy Returns

philarchive.org

Post reply on HN
Print view View original

271–280 of 437 posts

Re: AGI is Mathematically Impossible 2: When Entropy Returns

#271

Post by usrbinbash »

bubblyworld wrote: post #235

I find the mathematics in this paper a little incoherent so it's hard to criticise it on those grounds - but on a charitable read, something that sticks out to me is the assumption that AGI is some fixed total computable function from the fixed decision domain to a policy. AIs these days autonomously seek information themselves. Much like living things, they are recycling entropy and information to/from their environ…

> Much like living things, they are recycling entropy and information to/from their environment (the internet) at runtime.

3 Problems with that assumption:

a) Unlike living things, that information doesn't allow them to change. When a human touches a hotplate for the first time, it will (in addition to probably yelling and cursing a lot), learn that hotplates are dangerous and change its internal state to reflect that.

What we currently see as "AI" doesn't do that. Information gathered through means such as websearch + RAG, has ZERO impact on the systems internal makeup.

b) The "AI" doesn't collect the information. The model doesn't collect anything, and in fact can't. It can produce some sequence that may or may not cause some external entity to feed it back some more data (e.g. a websearch, databases, etc.). That is an advantage for technical applications, because it means we can easily marry an LLM to every system imaginable, but its really bad for the prospect of an AGI, that is supposed to be "autonomous".

c) The representation of the information has nothing to do with what it represents. All information an LLM works with, including whatever it is eing fed from th outside, is represented PURELY AND ONLY in terms of statistical relationships between the tokens in the message. There is no world-model, there is no understanding of information. There is mimicry of these things, to the point where they are technically useful and entice humans to anthropomorphise them (a BIIIG chunk of VC money hinges on that), but no actual understanding...and as soon as a model is left to its own devices, which would be a requirement for an AGI (remember: Autonomous), that becomes a problem.

Re: AGI is Mathematically Impossible 2: When Entropy Returns

#272

Post by 317070 »

ICBTheory wrote: post #259
vidarh wrote: post #180

Earlier quoted context omitted.

> Basically because physical laws obviously allow more than algorithmic cognition and problem solving. This is not obvious at all. Unless you can prove that humans can compute functions beyond the Turing computable, there is no basis for thinking that humans embody and physics that "allow more than algorithmic cognition". Your claim here also goes against the physical interpretation of the Church-Turing thesis. Witho…

No problem here is you proof - although a bit long: 1. THEOREM: Let a semantic frame be defined as Ω = (Σ, R), where Σ is a finite symbol set and R is a finite set of inference rules. Let Ω′ = (Σ′, R′) be a candidate successor frame. Define a frame jump as: Frame Jump Condition: Ω′ extends Ω if Σ′\Σ ≠ ∅ or R′\R ≠ ∅ Let P be a deterministic Turing machine (TM) operating entirely within Ω. Then: Lemma 1 (Symbol Contain…

> Let a semantic frame be defined as Ω = (Σ, R)

But if we let an AGI operate on Ω2 = (English, Science), that semantic frame would have encompassed both Newton and Einstein.

Your argument boils down into one specific and small semantic frame not being general enough to do all of AGI, not that _any_ semantic frame is incapable of AGI.

Your proof only applies to the Newtonian semantic frame. But your claim is that it is true for any semantic frame.

Re: AGI is Mathematically Impossible 2: When Entropy Returns

#273

Post by cess11 »

bubblyworld wrote: post #252
cess11 wrote: post #244

Earlier quoted context omitted.

I suspect there's a harsher argument to be made regarding "autonomous". Pull the power cord and see if it does what a mammal would do, or if it rather resembles a chaotic water wheel.

I think it would turn off, no shocker there. I'm not sure what you mean, can you elaborate? When I say autonomous I don't mean some high-falutin philosophical concept, I just mean it does stuff on it's own.

Right, but it doesn't. It stops once you stop forcing it to do stuff.

Re: AGI is Mathematically Impossible 2: When Entropy Returns

#274

Post by vidarh »

TheOtherHobbes wrote: post #234
vidarh wrote: post #229

Earlier quoted context omitted.

I'm not sure if this will help, but happy to elaborate further: The set of Turing computable functions is computationally equivalent to the lambda calculus, is computationally equivalent to the generally recursive functions. You don't need to understand those terms, only to know that these functions define the set of functions we believe to include all computable functions . (There are functions that we know to not b…

What program would a Turing machine run to spontaneously prove the incompleteness theorem? Can you prove such a program may exist?

Assuming the Church-Turing thesis is true, the existence of any brain now or in the past capable of proving it is proof that such a program may exist.

If the Church-Turing thesis can be proven false, conversely, then it may be possible that such a program can't exist - it is a necessary but not sufficient condition for the Church-Turing thesis to be false.

Given we have no evidence to suggest the Church-Turing thesis to be false, or for it to be possible for it to be false, the burden falls on those making the utterly extraordinary claim that they can't exist to actually provide evidence for those claims.

Can you prove the Church-Turing thesis false? Or even give a suggestion of what a function that might be computable but not Turing computable would look like?

Keep in mind that explaining how to compute a function step by step would need to contain at least one step that can't be explain in a way that allows the step to be computable by a Turing machine, or the explanation itself would instantly disprove your claim.

The very notion is so extraordinary as to require truly extraordinary proof and there is none.

A single example of a function that is not Turing computable that human intelligence can compute should be low burden if we can exceed the Turing computable.

Where are the examples?

Re: AGI is Mathematically Impossible 2: When Entropy Returns

#275

Post by ben_w »

cma wrote: post #187
moomin wrote: post #183

Earlier quoted context omitted.

I’m beginning to feel like the tests are part of the problem. Our intelligence tests are all tests of specialisation. We’ve established LLMs are part of the problem. Plenty of people who would fail a bar exam yet still know how many Rs there are in strawberry, could learn a new game just by reading the rules, know how to put up a set of shelves.

If you rarely got to see letters and just saw fragments of words as something like Chinese characters (tokens), could you count the R's in arbitrary words well? The bigger issue is LLMs still need way way more data than humans get tons what they do. But they also have many less parameters than the human brain.

> If you rarely got to see letters and just saw fragments of words as something like Chinese characters (tokens), could you count the R's in arbitrary words well?

While this seems correct, I'm sure I tried this when it was novel and observed that it could split the word into separate letters and then still count them wrong, which suggested something weird is happening internally.

I just now tried to repeat this, and it now counts the "r"'s in "strawberry" correctly (presumably enough examples of this specifically on the internet now?), but I did find it making the equivalent mistake with a German word (https://chatgpt.com/share/6859289d-f56c-8011-b253-eccd3cecee...):

  How many "n"'s are in "Brennnessel"?
But even then, having it spell the word out first, fixed it: https://chatgpt.com/share/685928bc-be58-8011-9a15-44886bb522...

Re: AGI is Mathematically Impossible 2: When Entropy Returns

#276

Post by vidarh »

somewhereoutth wrote: post #268
vidarh wrote: post #229

Earlier quoted context omitted.

I'm not sure if this will help, but happy to elaborate further: The set of Turing computable functions is computationally equivalent to the lambda calculus, is computationally equivalent to the generally recursive functions. You don't need to understand those terms, only to know that these functions define the set of functions we believe to include all computable functions . (There are functions that we know to not b…

> then the only way of showing that an artificial intelligence can not theoretically be constructed to at least meet the same bar is by showing that humans can compute more than the Turing computable. I would reframe: the only way of showing that artificial intelligence can be constructed is by showing that humans cannot compute more than the Turing computable. Given that Turing computable functions are a vanishingly…

Given that we know of no computable function that isn't Turing computable, and the set of Turing computable functions is known to be equivalent to the lambda calculus and equivalent to the set of general recursive functions, what is an immensely large hurdle would be to show even a single example of a computable function that is not Turing computable.

If you can do so, you'd have proven Turing, Kleen, Church, Goedel wrong, and disproven the Church-Turing thesis.

No such example is known to exist, and no such function is thought to be possible.

> Turing machines (and equivalents) are predicated on a finite alphabet / state space, which seems woefully inadequate to fully describe our clearly infinitary reality.

1/3 symbolically represents an infinite process. The notion that a finite alphabet can't describe inifity is trivially flawed.

Re: AGI is Mathematically Impossible 2: When Entropy Returns

#277

Post by ICBTheory »

yababa_y wrote: post #269
ICBTheory wrote: post #259

Earlier quoted context omitted.

No problem here is you proof - although a bit long: 1. THEOREM: Let a semantic frame be defined as Ω = (Σ, R), where Σ is a finite symbol set and R is a finite set of inference rules. Let Ω′ = (Σ′, R′) be a candidate successor frame. Define a frame jump as: Frame Jump Condition: Ω′ extends Ω if Σ′\Σ ≠ ∅ or R′\R ≠ ∅ Let P be a deterministic Turing machine (TM) operating entirely within Ω. Then: Lemma 1 (Symbol Contain…

Whoa there boss, extremely tough for you to casually assume that there is a consistent or complete metascience / metaphysics / metamathematics happening in human realm, but then model it with these impoverished machines that have no metatheoretic access. This is really sloppy work, I'd encourage you to look deeper into how (eg) HOL models "theories" (roughly corresponding to your idea of "frame") and how they can evo…

The claim isn’t that humans maintain a consistent metascience. In fact, quite the opposite. Frame jumps happen precisely because human cognition is not locked into a consistent formal system. That’s the point. It breaks, drifts, mutates. Not elegantly — generatively. You’re pointing to HOL-in-HOL or other meta-theoretical modeling approaches. But these aren’t equivalent. You can model a frame-jump after it has occurred, yes. You can define it retroactively. But that doesn’t make the generative act itself derivable from within the original system. You’re doing what every algorithmic model does: reverse-engineering emergence into a schema that assumes it. This is not sloppiness. It’s making a structural point: a TM with alphabet Σ can’t generate Σ′ where Σ′ \ Σ ≠ ∅. That is a hard constraint. Humans, somehow, do. If you don’t like the label “frame jump,” pick another. But that phenomenon is real, and you can’t dissolve it by saying “well, in HOL I can model this afterward.” If computation is always required to have an external frame to extend itself, then what you’re actually conceding is that self-contained systems can’t self-jump — which is my point exactly...

Re: AGI is Mathematically Impossible 2: When Entropy Returns

#278

Post by ICBTheory »

317070 wrote: post #272
ICBTheory wrote: post #259

Earlier quoted context omitted.

No problem here is you proof - although a bit long: 1. THEOREM: Let a semantic frame be defined as Ω = (Σ, R), where Σ is a finite symbol set and R is a finite set of inference rules. Let Ω′ = (Σ′, R′) be a candidate successor frame. Define a frame jump as: Frame Jump Condition: Ω′ extends Ω if Σ′\Σ ≠ ∅ or R′\R ≠ ∅ Let P be a deterministic Turing machine (TM) operating entirely within Ω. Then: Lemma 1 (Symbol Contain…

> Let a semantic frame be defined as Ω = (Σ, R) But if we let an AGI operate on Ω2 = (English, Science), that semantic frame would have encompassed both Newton and Einstein. Your argument boils down into one specific and small semantic frame not being general enough to do all of AGI, not that _any_ semantic frame is incapable of AGI. Your proof only applies to the Newtonian semantic frame. But your claim is that it i…

Yes, of course — if you define Ω² as “English + All of Science,” then congratulations, you have defined an unbounded oracle. But you’re just shifting the burden.

No sysem starting from Ω₁ can generate Ω₂ unless Ω₂ is already implicit. ... If you build a system trained on all of science, then yes, it knows Einstein because you gave it Einstein. But now ask it to generate the successor of Ω² (call it Ω³ ) with symbols that don’t yet exist. Can it derive those? No, because they’re not in Σ². Same limitation, new domain. This isn’t about “a small frame can’t do AGI.” It’s about every frame being finite, and therefore bounded in its generative reach. The question is whether any algorithmic system can exeed its own Σ and R. The answer is no. That’s not content-dependent, that’s structural.

Re: AGI is Mathematically Impossible 2: When Entropy Returns

#279

Post by bubblyworld »

usrbinbash wrote: post #271
bubblyworld wrote: post #235

I find the mathematics in this paper a little incoherent so it's hard to criticise it on those grounds - but on a charitable read, something that sticks out to me is the assumption that AGI is some fixed total computable function from the fixed decision domain to a policy. AIs these days autonomously seek information themselves. Much like living things, they are recycling entropy and information to/from their environ…

> Much like living things, they are recycling entropy and information to/from their environment (the internet) at runtime. 3 Problems with that assumption: a) Unlike living things, that information doesn't allow them to change. When a human touches a hotplate for the first time, it will (in addition to probably yelling and cursing a lot), learn that hotplates are dangerous and change its internal state to reflect tha…

It's not really an assumption, it's an observation. Run an agentic tool and you'll see it do this kind of thing all the time. It's pretty clear that they use the information to guide themselves (i.e. there's an entropy reduction there in the space of future policies, if you want to use the language of the OP).

> Unlike living things, that information doesn't allow them to change.

It absolutely does. Their behaviour changes constantly as they explore your codebase, run scripts, question you... this is just plainly obvious to anyone using these things. I agree that somewhere down the line there is a fixed set of tensors but that is not the algorithm. If you want to analyse this stuff in good faith you need to include the rest of the system too, including it's memory, context and more generally any tool it can interact with.

> The "AI" doesn't collect the information.

I really don't know how to engage on this. It certainly isn't me collecting the information. I just tell it what I want it to do at a high level and it goes and does all this stuff on its own.

> There is no world-model, there is no understanding of information.

I'm also not going to engage on this. I could care less what labels people assign to the behaviour of AI agents, and whether it counts as "understanding" or "intelligence" or whatever. I'm interested in their observable behaviour, and how to use them, not so much in the philosophy. In my experience trying to discuss the latter just leads to flame wars (for now).

Re: AGI is Mathematically Impossible 2: When Entropy Returns

#280

Post by vidarh »

ICBTheory wrote: post #277
yababa_y wrote: post #269

Earlier quoted context omitted.

Whoa there boss, extremely tough for you to casually assume that there is a consistent or complete metascience / metaphysics / metamathematics happening in human realm, but then model it with these impoverished machines that have no metatheoretic access. This is really sloppy work, I'd encourage you to look deeper into how (eg) HOL models "theories" (roughly corresponding to your idea of "frame") and how they can evo…

The claim isn’t that humans maintain a consistent metascience. In fact, quite the opposite. Frame jumps happen precisely because human cognition is not locked into a consistent formal system. That’s the point. It breaks, drifts, mutates. Not elegantly — generatively. You’re pointing to HOL-in-HOL or other meta-theoretical modeling approaches. But these aren’t equivalent. You can model a frame-jump after it has occurr…

> It’s making a structural point: a TM with alphabet Σ can’t generate Σ′ where Σ′ \ Σ ≠ ∅

This is trivially false. For any TM with such an alphabet, you can run a program that simulates a TM with an alphabet that includes Σ′.

Post reply on HN