Superintelligence: The Idea That Eats Smart People (2016)

The first two are collections of short stories. You can find something from the Cyberiad online if you look for the pdf- Trurl's electronic bard is sort of spot on in this thread.

That argument is totally nonsense. A hundred years ago, you could argue that humans cannot invent an internet, because over billions of years (minus one hundred), it's never happened.

There's a more rigorous way to argue for the claim in question, which debuted in my doctoral dissertation. Or, not quite the exact claim in question, but rather the following claim: that an idealized mechanical knowing agent cannot create a more intelligent idealized mechanical knowing agent than itself (at least, not one which it can actually trust). The idea behind the proof is quite simple, but requires familiarity with computable ordinal numbers. If knowing agent X creates knowing agent Y, then presumably X knows Y's sourcecode. Thus, X can infer a sourcecode for the list of all naturals n such that Y knows "n is the code of a computable ordinal". If X actually trusts Y, then X knows that whenever Y knows "n is the code of a computable ordinal", then n really is the code of a computable ordinal. Combining these, X can infer the ordinality of the ordering obtained by concatenating together all the ordinals which Y knows are ordinals. Thus, X knows a larger computable ordinal than all the computable ordinals Y knows. Which, depending on your definition of intelligence, arguably suggests X is more intelligent than Y.

I would argue that a "human level" intelligence that can think thousands or millions of times faster than a human with a vastly larger working memory (and, sometimes it is argued, can be trivially scaled to work faster / more efficiently given more hardware) is in fact already a super-intelligence.

If you want to argue for intelligence explosion given a "human level" intelligence, I say start with an AI black box wherein if you give a basic algebra problem to an ordinary human at the same time as you give it to the black box, they will think for about 30 seconds and then give an answer that is half likely to be wrong.

If you want to argue from a "human level" intelligence, you need to acknowledge that throwing more hardware doesn't inherently make it smarter. You can throw five ordinary humans in a room and give them a simple algebra problem, it doesn't mean they will solve it five times faster. Likely they will quickly decide amongst themselves who is best and math, and that one person will then give their answer to the algebra problem, which should be slightly more likely to be the correct answer (and possibly a slightly slower response) than if you just tried with one ordinary human.

The Einstein cat example is about trying to coerce a non-compliant entity that you cannot communicate with into complying. The Stephen Hawking cat example was about trying to model the behaviour of the non-compliant entity in order to persuade it to comply. The use of a cat as the agent in both cases alludes to the behaviour that people dealing with cats frequently experience.

I'm personally not terribly worried, as I think we have a ways to go before creating human-like general intelligence, let alone super intelligence. It is also not clear at all to me that an exponential increase in intelligence is likely or feasible after some threshold. Still, treading carefully seems prudent. There are some serious risks, and even if the likelihood appears low, there are a lot of unknowns.