Likelihood of discontinuous progress around the development of AGI
1–10 of 111 posts
Re: Likelihood of discontinuous progress around the development of AGI
#2That feeling an article will help more confusing people than enlightening them, just reading it's title...
Re: Likelihood of discontinuous progress around the development of AGI
#3The counterargument on the last one (Human-competition threshold) seems weak (and simply wrong) to me. In related link argument for wide range of human abilities rests basically on comatose humans being unable to do anything at all and mutation-adds-random piece-to-machine metaphor.
Mentally-impaired human intelligence level is enough to replace some of the workforce, combined with some actuators and perception abilies of average human.
Re: Likelihood of discontinuous progress around the development of AGI
#4> This argument seems weak to us currently, but further research could resolve these questions in directions that would make it compelling:
> Are individual humans radically superior to apes on particular measures of cognitive ability? What are those measures, and how plausible is it that evolution was (perhaps indirectly) optimizing for them?
Yes, clearly. All the measures defined in this article are arbitrary (e.g. vehicular land speed), and so let's propose an arbitrary one for intelligence: Ability to prove mathematical theorems. Humans have proven many. Apes have proven zero. That is an extreme discontinuity. We can propose many others, of course: complex language development, building skyscrapers, landing on the moon, etc..
> How likely is improvement in individual cognitive ability to account for humans’ radical success over apes? (For instance, compared to new ability to share innovations across the population)
How does human cognitive ability being over-rated counter the discontinuity in intelligence development? Is the argument that our success is due to some other characteristic than intelligence, and so our intelligence is not really that much greater than the apes? If so, that's just a restatement of point 1.
Re: Likelihood of discontinuous progress around the development of AGI
#5We know that for most problems, a group of people tends to be better at problem solving than an individual [1]. Even if AI technology only reaches human-level and does not exceed it, continually increasing efficiency would make AI smarter than any small groups of humans and immense bandwidth relative to human communications would make it more effective than any large human organizations.
[1] https://aiimpacts.org/coordinated-human-action-example-super...
In addition, if AI posesses sufficient computing resources, which will certainly become available in the next few decades if not already, it will have inherent strong advantages like serial computation speed and memory size that exceed any human brains. So the real barrier for AGI is software and not hardware.
If AGI software is developed before we have sufficient hardware to run it at the human brain speed, then it will become more capable at the rate of hardware we can put into use, which is likely exponential given how parallelized the human brain appears to be.
The major counterargument I find most convincing regarding outsized impact of exploding intelligence is that many problems are exponentially hard (or harder than that) and thus exponential intelligence can only make linear or sublinear progress on them.
However, linear or even sublinear progress may still lead to quite drastic changes in the world. If an organization can marginally predict stock price movements better than the rest of the world, in a few decades it will accumulate great resources and power. The same is true for many other important domains.
Re: Likelihood of discontinuous progress around the development of AGI
#6We know that for most problems, a group of people tends to be better at problem solving than an individual [1]. Even if AI technology only reaches human-level and does not exceed it, continually increasing efficiency would make AI smarter than any small groups of humans and immense bandwidth relative to human communications would make it more effective than any large human organizations. [1] https://aiimpacts.org/coo…
This also assumes intelligence is the limiting factor for solving many problems. I suspect information and computation is probably the larger factor for most major issues. The smartest player possible would still lose poker to someone that can read their hand.
Re: Likelihood of discontinuous progress around the development of AGI
#7See also: OpenAI's Paul Christiano wrote on the same topic — https://www.lesserwrong.com/posts/AfGmsjGPXN97kNp57/argument...
(cf. https://intelligence.org/files/IEM.pdf for some of the arguments being argued against)
Re: Likelihood of discontinuous progress around the development of AGI
#8> This argument seems weak to us currently, but further research could resolve these questions in directions that would make it compelling: > Are individual humans radically superior to apes on particular measures of cognitive ability? What are those measures, and how plausible is it that evolution was (perhaps indirectly) optimizing for them? Yes, clearly. All the measures defined in this article are arbitrary (e.g.…
>Yes, clearly. All the measures defined in this article are arbitrary (e.g. vehicular land speed), and so let's propose an arbitrary one for intelligence: Ability to prove mathematical theorems. Humans have proven many. Apes have proven zero. That is an extreme discontinuity. We can propose many others, of course: complex language development, building skyscrapers, landing on the moon, etc..
That's at least partially explained culturally - pressure to go attent university leads to more exposure to mathematics and stuff.
Also, how do you actually know that gorillas haven't proven mathematical formulas? I mean, they don't have writing so perhaps there were gorilla mathematicians before, they just weren't recorded.
Re: Likelihood of discontinuous progress around the development of AGI
#9We know that for most problems, a group of people tends to be better at problem solving than an individual [1]. Even if AI technology only reaches human-level and does not exceed it, continually increasing efficiency would make AI smarter than any small groups of humans and immense bandwidth relative to human communications would make it more effective than any large human organizations. [1] https://aiimpacts.org/coo…
This also assumes intelligence is the limiting factor for solving many problems. I suspect information and computation is probably the larger factor for most major issues. The smartest player possible would still lose poker to someone that can read their hand.
I agree that information is crucial to achieve a 'win' for many goals. Given today's amount of information on the Internet as well as electronic money and access to most officials, barring some sort of inviolable built-in moral core, an AGI would be able to use any methods, overt and covert, direct and cunning, technical and social, to achieve its information goals. [1]
Since an AGI can copy itself and be available at a multitude of access points at once and those copies can often communicate via extremely fast channels, it is human organizations that would be at an information disadvantage.
[1] This also assumes that the AGI does not have the will nor the capability to change its own moral core. I think an AGI will possibly be capable of changing its own core, so a much more reliable safeguard is to make sure that it does not want to change it.
Re: Likelihood of discontinuous progress around the development of AGI
#10It's an interesting read, but perhaps it only succeeds in pointing out the gaps in our knowledge. When I read their counterargument to the possibility of intelligence explosion:
"Positive feedback loops are common in the world, and very rarely move fast enough and far enough to become a dominant dynamic in the world."
the idea that immediately comes to mind is the Harmless Supernova Fallacy, described on this (obnoxiously JavaScript-dependent) site:
https://arbital.com/p/harmless_supernova/
Knowledge of this fallacy is a mental tool I have found quite useful, as it seems to be a type of fallacy that is easy to make by accident. To be fair, the reasoning in the article may not quite reach the level of a fallacy, but the intelligence explosion section ends saying effectively this:
"we think the intelligence explosion argument could be strong if strong reason is found to expect an unusually fast and persistent feedback loop [i.e. an intelligence explosion]"
which sounds like a classic case of Begging the Question: