Why general artificial intelligence will not be realized

The point is that humans and computers operate differently. The human approach is based on adaptive experiential heuristics.

The computer approach is based on explicit formalism. (Even in neural networks, there's still a formal model. It's just made of weightings instead of logic paths.)

The epistemology of these approaches is completely different. The problem isn't getting a computer to ride a bike, it's getting a computer to learn to ride a bike how a human learns.

Why would anyone do this? Because adaptive experiential heuristics are far more flexible and generalisable than explicit formalisms. And - it suggests here - you can't have real AGI without them.

So the problem then becomes unpicking what "adaptive" and "experiential" really mean. Both rely on huge accumulations of tacit knowledge and tacit motivations.

If this isn't obvious, consider that a human child will learn how to ride a bike and then go and have a lot of fun with it. An ideal bike-riding computer doesn't even have a concept of fun.

The human experience of fun is a complex system of experimentation, exploration, reward, and challenge, combined with physical, emotional, and mental correlates.

This matters because play in childhood helps develop the heuristics that adults use for problem solving, and for personal motivation and satisfaction.

Even more simply, the problem is the difference between building a workable but dumb bike riding machine and building a machine that will improvise bike riding as a goal for itself, will "enjoy" the experience, and will generalise from that to mastery of other domains.

What I think what the article is talking about, is the idea that AGI must be embodied. That a widely unrecognized aspect of HI is its embodiment (read The Power of not thinking), and that without a body machines cannot be in the world, and therefore cannot have AGI.

I tend to think however that machine AGI will be possible even without putting them in robot bodies. They already have their own world, the internet. Just get them to explore that with a virtual analogy of a body. Worth a try anyway.

The tacit knowledge description only talks about the acquisition process of the knowledge, not that the knowledge itself is outside bounds of rationality or the physical world. By the same token when you claim "humans have intelligence that is impossible to express through reason or codification" the entire argument hinges on the actual meaning of impossible. Is it impossible in itself, or ever, because human intelligence is non-reason based or are we using a meaning of impossible that at this day and age the process of doing that is still intractable for all intents and purposes. If you're claiming the former, that itself is such a strong claim it requires its own strong proof. If it's the latter, well we have been developing psychotechnologies for several millennia to be able to express ourselves and our cognitive processes and getting better at it, you just need to be patient.

If you found a working x86 chip in the wild, but all documentation and knowledgeable people were wiped out for some reason, I bet the process of finding out how that x86 worked would look quite similar. It wouldn't make x86 otherworldly or in need of a soul.

As the article hints at, his line of argument was a completely valid criticism of AI based on a set of rules written with symbolic logic. He arrived at this conclusion after studying Heidegger's concept of dasein. The best way to describe dasein is through a classic example [1]:

"...the hammer is involved in an act of hammering; that hammering is involved in making something fast; and that making something fast is involved in protecting the human agent against bad weather. Such totalities of involvements are the contexts of everyday equipmental practice. As such, they define equipmental entities, so the hammer is intelligible as what it is only with respect to the shelter and, indeed, all the other items of equipment to which it meaningfully relates in Dasein's everyday practices. "

In Heidegger's mind, meaning is distributed across the web of interrelationships between objects and their various uses and ideas to humans. Intelligence was thus a process of knowing those relationships, and being a part of them. Being-in-the-world (dasein is a German word related to 'being') is a result of us as humans being 'thrown' into the web of meaning, in fact we are born already finding ourselves in it.

Dreyfus' objections to AI stemmed from the idea that computers are thrown into the world differently from us. They are the hammer in the anecdote, and not the human. During the class, we argued a lot with him about whether humans are the only creatures that have 'being-in-the-world'. We asked about the idea of the soul, and why humans are unique in this paradigm. My feeling after the end was that his entire philosophy comes directly from Heidegger. And since Heidegger didn't mention animals, or the lack of souls, his conclusion was "dunno".

Related to this, Dreyfus had an understanding of physics that was quite antiquated and classical. One example was the concept of time. According to Dreyfus, the physicists' notion of time was a series of discrete observations along a timeline, whereas humans experience time in stretches and long segments. I remember telling him that the notion of a single instant of time is poorly defined in quantum physics, and that we do recognize that every event has some time uncertainty. I remember asking him why we couldn't model human experience with some complicated function. For him, everything came back to Heidegger. To him, we physicists were being too reductionist, and there is indeed something about humans that cannot be described using physics. He stopped shy of calling it a soul, but it was essentially that.

[ETA: I now remember talking to him about Conway's game of life, and how a simple set of rules could result in complex, often hard to model behavior. My point was emergent systems exist within physics, and the way we describe them is different from single particles. His reply suggested that he was certain that human experience wasn't just 'emergent' - it was fundamentally different from anything in physics, no matter what.]

The class was on physics vs philosophy, and we disagreed. I don't think he really understood what I was trying to tell him. This was in 2013, before most of the deep learning revolution, but I think he would have the same objections with what we have now. Here are two possible directions we can go from here:

1) Argue with continental philosophers about reductionism and whether humans have a unique essence that cannot be modeled.

2) Understand Heidegger's and Dreyfus' thoughts about being and time, and drive AI research in better directions.

I prefer (2), because I already tried (1) with Dreyfus and it wasn't successful or productive.

I think understanding and modeling the graph of inter-relationships between objects and humans is exactly where we need to improve when it comes to AGI. It's probably going to need a good degree of embodiedness, an idea of a computer being thrown-into-the-world. Dreyfus would tell us that it's fundamentally impossible, and that we should all just give up on it. I think he came to the wrong conclusion. What I get from Heidegger is not "AGI is impossible" but rather "hey, this is what we should be worrying about. This is how humans see the world".

TLDR: Dreyfus had some good ideas about AI. I think it's extremely insightful to pay attention his thoughts. Don't waste time worrying about his arguments against physicalism.

[1] https://plato.stanford.edu/entries/heidegger/