David Deutsch On Artificial Intelligence

Say AGI is a computer machine and/or algorithm that's capable of "creatively" building a smarter version of itself. Then (here's the proof) say we did build a machine that was capable of building a smarter version of itself, then that machine would technically be a perpetual motion machine, and therefore a violation of the first fundamental law of thermodynamics: "In all cases in which work is produced by the agency of heat, a quantity of heat is consumed which is proportional to the work done; and conversely, by the expenditure of an equal quantity of work an equal quantity of heat is produced." (Rudolf Clausius, 1850)

Or assuming that AGI is just "in the software", the heat produced by the computation would continually increase, as an ever more complicated algorithm/computation is formulated, and therefore violate the first law of thermodynamics -- again. Assuming a more "intelligent" computation consumes more heat/energy.

Wait, we don't know that for sure, yet.

One way to implement the 'remembering' of the ball position is to run a forward model of the trajectory in your head.

Your brain has specialized hardware for running forward models (short-term simulations) of physical systems. You need these to control your body precisely in the face of delays in the feedback from your limbs.

A nice story - it's a just-so story, but a nice one - is that we repurpose this simulation hardware to simulate systems we observe outside our bodies. Expert ball-catchers have a very good simulation, beginners have a bad one. If you're already an expert, you can get even better by running your simulation off-line: this may be why experts can practice by visualizing their golf swing or swim stroke but newbies can not improve much that way.

There is some physical evidence that the off-line model stuff is real, as when you imagine yourself walking, or watch someone else walking, some of the same parts of your motor cortex are active as when you are actually walking.

My very favourite part of this explanation trail is as follows: animals got very good at modelling the behaviour of other animals, in order to predict what they will do next: Eventually, they/we started re-using the same systems to understand and explain our own behaviour into the future. This is (at least part of) what we call consciousness.

Now, you may or may not buy this, but it's rather elegant as a hypothesis.

But we're kind of long past the point where just any old random slurry of chemicals is going to get means-tested in the great arena of life. Life is not finding the right set of chemicals to combine, life is a specific set of chemicals and the right orientations of different copies of those chemicals.

So I think we're at a point were binary code, the instructions to run on the processor, is akin to chemicals in the physical world. We try to treat them like DNA, but you can't just toss a bunch of chemicals in a bucket at random and expect life to come out. 100 billion times out of 100 billion times, random chemicals in a bucket makes you nothing close to life. Ultimately, chemicals are at the core, but they aren't sufficient. The right chemicals are needed, and they interact in such a way that infinite variation is the result.

And DNA is a code--a deterministic, exceedingly discrete code. Yet somehow (hand waiving), from such arises the non-deterministic, comparatively-infinite variability of human behavior. So in that sense, I don't think he's necessarily correct that AGI is a "different type of program than we've ever programmed."

But all that is just to create a system that is intelligent, it's not intelligent itself. It's a road, not a destination. Living things, on the other hand, have goals and try to achieve them. Not just have goals, but generate goals. Create it's own notions of what to do and how to do them and why the doing of it is important.

You touch fire, your hand recoils, because in you is a system for detecting potential damage and the understanding that damage is not something you want on your docket. Computer touches fire, computer recoils, because in it is a system for detecting potential damage and YOUR understanding that damage is not good. The computer didn't conclude on its own that damage was bad. It never had the sense that it existed. And this isn't even an "intelligent" response, this one is merely instinct.

So I think the big, missing question in AGI is, "what could a computer want?" We could program a computer to have certain goals, but that is not the same thing as a computer sitting around and saying, "hey, you know what? Let's go to the beach this weekend." We foist our own goals on the computer and instruct it on how to understand those goals, and are disappointed when it fails to get the point of the goals at all and sits there blinking at us. How can you ever hope to have an intelligent computer if it isn't intelligent on its own terms?

IDK, I am rambling. Would you be intelligent or have any hope of becoming intelligent if you didn't create your own designs on your future, couldn't perceive anything to test your actions against your desires for the future, and had no means of your own to ever come about correcting these issues? It just seems like the only way AGI will happen is through something incredibly simple that allows a computer to put its own parts together, see the result, and arbitrarily evaluate it.