Superintelligence: The Idea That Eats Smart People

> so blithely assume so easily that there is no quantum effect in the brain structure.

He is following the mainstream opinion on this view, so does not have to justify it in detail. Almost nobody subscribes to Penrose's proposal.

> Our best approximations of intelligence are analogous to the comparison between sewing by hand vs the mechanics of a sewing machine, only the sewing machine remains far inferior (at least for now).

There's a blithe assumption of your own, which you don't attempt to justify. Newell and Simon's physical symbol system hypothesis is still taken seriously by mainstream AI, and it has the opposing view: that intelligence is a computational process, and thus we understand many fundamental things about it already.

I'm not taking a position here, just clarifying what the dominant positions are right now.

Also, sewing machines are better at sewing than 99% of people by any metric I can think of, and faster than 100% of people. They also do probably >90% of all the sewing on the planet. So I don't think your analogy does the work you intended.

Pretty poorly argued. The AI alarmists simply argue that if the super-intelligence's objective isn't defined correctly, the super-intelligence will wipe us out as a mere consequence of pursuing its objective, not that the super-intelligence will try to conquer us in a specific way like Einstein putting his cat in a cage. The alarmists' argument is analogous to humans wiping out ecosystems and species by merely doing what humans do and not by consciously trying to achieve that destruction. Many of the author's arguments stem from this fundamental mistake.

There 's nothing suggesting that neurons are not deterministic, in fact they are simulated pretty well with some cable theory and HH models. https://en.wikipedia.org/wiki/Hodgkin%E2%80%93Huxley_model

"Put another way, this is the premise that the mind arises out of ordinary physics... If you are very religious, you might believe that a brain is not possible without a soul. But for most of us, this is an easy premise to accept."

The thing that irks me about this is how it reinforces a common (and in my opinion, false) dichotomy: either you believe the mind is explicable in terms of ordinary physics or you believe in a soul and are therefore religious. I feel like there should be a third way, one that admits something vital is missing from the physicalist picture but doesn't make up a story about what that thing is. There is a huge question mark at the heart of neuroscience -- the famed Explanatory Gap -- and I think we should be able to recognize that question mark without being labeled a Supernaturalist. Consciousness is weird!

"Not many people know that Einstein was a burly, muscular fellow. But if Einstein tried to get a cat in a carrier, and the cat didn't want to go, you know what would happen to Einstein. He would have to resort to a brute-force solution that has nothing to do with intelligence, and in that matchup the cat could do pretty well for itself."

This seems, actually, like a perfect argument going in the other direction. Every day, millions of people put cats into boxes, despite the cats not being interested. If you offered to pay a normal, reasonably competent person $1,000 to get a reluctant cat in a box, do you really think they simply would not be able to do it? Heck, humans manage to keep tigers in zoos, where millions of people see them every year, with a tiny serious injury rate, even though tigers are aggressive and predatory by default and can trivially overpower humans.