...it is simply not true that knowledge comes from extrapolating repeated observations. Nor is it true that ‘the future is like the past’, in any sense that one could detect in advance without already knowing the explanation.
This is obviously untrue. You could train a machine learning algorithm to count easily just by showing it examples of numbers. This is how we teach human children to count, by showing them examples. Not giving them some magical "explanation".
And he goes on to base the rest of his argument on this point.
>Some people are wondering whether we should welcome our new robot overlords. Some hope to learn how we can rig their programming to make them constitutionally unable to harm humans (as in Isaac Asimov’s ‘laws of robotics’), or to prevent them from acquiring the theory that the universe should be converted into paper clips (as imagined by Nick Bostrom). None of these are the real problem. It has always been the case that a single exceptionally creative person can be thousands of times as productive — economically, intellectually or whatever — as most people; and that such a person could do enormous harm were he to turn his powers to evil instead of good.
Einstein's brain was only slightly different than any other human's, even the dumbest village idiot. Humans themselves are only slightly different than chimpanzees.
So yes, an AI with a brain of entirely different architecture, running on computers millions of times faster and several times larger than a human brain, would indeed be pretty concerning.