Ray Kurzweil does not understand the brain

I think that is a very original use of the word 'limited', limited in this case holds enough room for random chance to come up with human beings.

For all practical purposes that 'limited' might as well be unlimited.

So the real question then becomes does biology tolerate working on an approximation of the underlying physics and does that simulated biology still have the ability to exhibit intelligence. I think the first is a maybe, the second a yes but I couldn't give you any reasons why other than that it might be that our biology needs 'its' physics to operate and that probably anything Turing complete has the potential to exhibit intelligence, regardless of whether or not we find a way to achieve that.

Yes, we're in full agreement on this, especially the part about Kurzweil - I don't think he advances the science of this at all, and his poppy PR approach rubs me the wrong way, for sure, as do many of his specific ideas.

In particular, I think it's a terrible idea to hinge our hopes for AI on the idea that we'll be able to carry out a perfect enough simulation of a full brain to call it "done". Not only does it seem to be a pipe dream at present, depending on a whole bunch of technological advances that may not come for a long time, it would be almost completely inextensible and teach us very little - once we had such a simulation running, it's unlikely that we'd understand enough about how to modify it to improve upon it.

I don't quarrel with the idea that DNA encodes the brain extremely indirectly at all, including a substantial development process that depends on a lot of things other than the pure genetic code, and I also agree that the dynamics of a brain are very dependent on the details of all sorts of cells, not just an oversimplified logical representation of them. Where I think we start to diverge is that I have issues with the idea that the particulars of any of those processes can somehow "piggy-back" any non-trivial amount of functionality into the system that's not coming from the genetic code.

Here's the clearest way, I think, to state my information-content claim: if we were to randomly change the details of the way neurons function in detail (still leaving them with the same highest-level capabilities), and randomly change the way brain development happens, and randomly change all steps of entire process that leads from DNA->brain (again, subject to the restriction that the whole thing still works), and even - I daresay - randomly change the laws of physics, then...

...if all of these changes still permitted us to write down any string of DNA (or whatever our new version of DNA was) that ultimately resulted in the growth of an intelligent human brain, the amount of DNA required would, on average, be pretty darn close to the amount that we see used today. And that has implications for other implementations of the same logic, namely that if we could find some way to optimize towards whatever solution evolution has found, we could probably end up with a shorter code for it than evolution has because we can allow all parts of the DNA->brain process to optimize for compression whereas the real physical process is largely frozen against change.

That's the sense in which I think Kurzweil has a reasonable estimate, nothing more, nothing less; where he takes it from there is another story altogether. :)

A lot of these questions are completely unanswered. We do know at least that the brain is extremely resilient to changes in chemistry and can work quite well even in the face of extensive damage, which is an indication that we might be surprised how little of the overall arrangement is actually necessary to keep it working properly.

We need to make sure that we're being careful with our language, too: the complexity necessary to specify any brain is a whole lot lower than the complexity to specify one particular brain. In AI the goal is the former, but for the latter, we really do have to worry about each strand in the carpet. An AI researcher might not give a rat's ass about reproducing your memories, and will be more than happy to construct something that can form any memories; on the other hand, to you, your memories (and the detailed wiring inside your head, which we might be able to alter substantially without "breaking" the brain) are vitally important.

Please, if you've subjected yourself to that nonsense, cleanse your palette with The Road To Reality - math/physics is where Penrose shines, and that book is him at his brightest.