Earlier quoted context omitted.
In fairness to Kurzweil, he isn't talking about a simple chatbot. That's a pretty big difference. But beyond that, there's an argument that there's no clear distinction between a simulation of a person and the person. If you want to get into this, start out by telling me the difference between A) a new sorting algorithm you wrote vs B) a simulation of the new sorting algorithm. In particular, why B isn't also running…
The problem with the analogy to a sorting algorithm is that a sorting algorithm is obviously computable. What if there's something like protein folding that has O(2^n) complexity involved in being a person that the body does trillions of times a second but is impossible for any size computer to do for large N?
But anyway: I guess its possible human biological hardware does something difficult to simulate on a computer (at least one similar to our current architectures.)
Given a super advanced intelligence, though, I doubt adding hardware that does protein folding (or the protein folding analogue operation) fast - in biology if necessary - is going to be a major obstacle.
And that's if it can't come up with an algorithm that does an equivalent operation, that runs well on whatever hardware it already has.