A Reply to François Chollet on Intelligence Explosion
intelligence.org
A Reply to François Chollet on Intelligence Explosion
1–10 of 89 posts
Re: A Reply to François Chollet on Intelligence Explosion
#2The more I read about this, the more I think we should stick to narrow AI. If you don't want your cancer research bot to take over the world, don't give it general reasoning capability. Goal alignment seems incredibly fragile in comparison.
Re: A Reply to François Chollet on Intelligence Explosion
#3Something to note about the "we're on an exponential (or at least more-than-linear) increase in technology" arguments: They generally need to reach back quite a long time to sound convincing.
Yudkowsky makes two arguments about how fast technology or society is evolving: in one he chooses 500 years ago, 1517. In another, he talks about "the last 10,000 years."
In contrast, Chollet compares 1900-1950 with 1950-2000.
I agree, we've changed the world in big ways compared to 1517 or 8000 BC. But if we're on a more-than-linear increase, then we should be seeing more and more technological growth in very recent timeframes, not needing to reach back centuries or millennia.
In fact, if you consider a y = log(x) or y = sqrt(x) function, those more closely fit a narrative of "If you look back a long time, things seem almost crazily changed, but if you look into recent history, it looks slower" much better than a y = x^2 or y = e^x function.
Re: A Reply to François Chollet on Intelligence Explosion
#4The more I read about this, the more I think we should stick to narrow AI. If you don't want your cancer research bot to take over the world, don't give it general reasoning capability. Goal alignment seems incredibly fragile in comparison.
by what means will a cancer research bot take over the world? it doesn't matter how smart you are if you don't have the necessary means to do something. i think it's a fantasy, something that people who imagine themselves to be very intelligent have latched onto-- the idea that their best quality is the best quality.
Re: A Reply to François Chollet on Intelligence Explosion
#5The more I read about this, the more I think we should stick to narrow AI. If you don't want your cancer research bot to take over the world, don't give it general reasoning capability. Goal alignment seems incredibly fragile in comparison.
by what means will a cancer research bot take over the world? it doesn't matter how smart you are if you don't have the necessary means to do something. i think it's a fantasy, something that people who imagine themselves to be very intelligent have latched onto-- the idea that their best quality is the best quality.
Well, the general AI doomsday fundamentalist argument proceeds as follows: you tell an AI to cure cancer. It can’t, so it spends some time recursively improving itself, then it finds out that the cost of curing cancer is C, but the cost of killing all humans (and therefore indirectly curing cancer) is C’, where C’ If you’re a smarty pants you tell the AI to cure cancer AND not kill all humans. But because the AI is so smart it comes up with something no human would have ever thought of, like putting all humans in eternal cryostasis, thereby keeping them alive AND eradicating cancer. No matter what you do, the AI will outsmart you because recursive-self-improvement, and humanity dies.
That’s what Elon Musk, Stephen Hawking, and others are worried about.
Re: A Reply to François Chollet on Intelligence Explosion
#6The more I read about this, the more I think we should stick to narrow AI. If you don't want your cancer research bot to take over the world, don't give it general reasoning capability. Goal alignment seems incredibly fragile in comparison.
by what means will a cancer research bot take over the world? it doesn't matter how smart you are if you don't have the necessary means to do something. i think it's a fantasy, something that people who imagine themselves to be very intelligent have latched onto-- the idea that their best quality is the best quality.
Presumably by getting people to take a drug with unexpected side effects? That doesn't seem particularly likely, but I don't think it's wrong to be paranoid about this. Defense in depth. Don't give the researcher the means to take arbitrary actions, and don't give it general reasoning ability.
Protecting against narrow AI will keep us plenty busy. Consider a narrow-AI penetration tester that falls into the wrong hands. Protecting against that sort of threat also helps protect against general AI threats.
Re: A Reply to François Chollet on Intelligence Explosion
#7Something to note about the "we're on an exponential (or at least more-than-linear) increase in technology" arguments: They generally need to reach back quite a long time to sound convincing. Yudkowsky makes two arguments about how fast technology or society is evolving: in one he chooses 500 years ago, 1517. In another, he talks about "the last 10,000 years." In contrast, Chollet compares 1900-1950 with 1950-2000. I…
This should be seen in the context of the article Yudkowsky is replying to. Collet's position is that runaway AGI is impossible, and in support of that, he claims that it is impossible for AI technology to grow super-linearly. In his refutation, Yudkowski has merely to show that it is not beyond the bounds of possibility.
Of course, insofar as he is also putting forward the view that the risk is significant, he is also putting forward the opinion that it is quite plausible - at least plausible enough to take the scenario seriously. That argument can survive occasional stalls in the rate of technological advance, if that is indeed what we are seeing.
Re: A Reply to François Chollet on Intelligence Explosion
#8Something to note about the "we're on an exponential (or at least more-than-linear) increase in technology" arguments: They generally need to reach back quite a long time to sound convincing. Yudkowsky makes two arguments about how fast technology or society is evolving: in one he chooses 500 years ago, 1517. In another, he talks about "the last 10,000 years." In contrast, Chollet compares 1900-1950 with 1950-2000. I…
Just for funsies (but may answer your question): every (differentiable) function is linear at a small enough time scale. This includes e^x or pretty much every function you listed.
Re: A Reply to François Chollet on Intelligence Explosion
#9Earlier quoted context omitted.
by what means will a cancer research bot take over the world? it doesn't matter how smart you are if you don't have the necessary means to do something. i think it's a fantasy, something that people who imagine themselves to be very intelligent have latched onto-- the idea that their best quality is the best quality.
Well, the general AI doomsday fundamentalist argument proceeds as follows: you tell an AI to cure cancer. It can’t, so it spends some time recursively improving itself, then it finds out that the cost of curing cancer is C, but the cost of killing all humans (and therefore indirectly curing cancer) is C’, where C’ If you’re a smarty pants you tell the AI to cure cancer AND not kill all humans. But because the AI is s…
I think the question is, and the one I have too, is why would such AI have any ability to do anything beyond output a Cure Cancer solution to a terminal? Why does the AI need to be the one to implement its derived solution?
The AI has a good think and comes up with a solution of killing all humans. The researchers read the printed report of the solution and decide against implementing it, tweak parameters, and ask the AI to take another go at it.
Re: A Reply to François Chollet on Intelligence Explosion
#10Earlier quoted context omitted.
by what means will a cancer research bot take over the world? it doesn't matter how smart you are if you don't have the necessary means to do something. i think it's a fantasy, something that people who imagine themselves to be very intelligent have latched onto-- the idea that their best quality is the best quality.
Well, the general AI doomsday fundamentalist argument proceeds as follows: you tell an AI to cure cancer. It can’t, so it spends some time recursively improving itself, then it finds out that the cost of curing cancer is C, but the cost of killing all humans (and therefore indirectly curing cancer) is C’, where C’ If you’re a smarty pants you tell the AI to cure cancer AND not kill all humans. But because the AI is s…
The concept is similar to the way a corporation given just the goal of increasing shareholder value will shit on developing countries with a Bhopal disaster or Niger Delta oil spills or opium wars and simply leave the country when it tries to enact penalties. Or the way a cigarette company or coal company will use politics, religion, and disinformation to protect its business against people affected by its products.
Incentives have to be very carefully aligned even for human level intelligences to prevent them from causing mass death and misery. Superhuman intelligences will be even better at achieving their goals, so the problem will only get worse.