Likelihood of discontinuous progress around the development of AGI • Hacker News Board

Re: Likelihood of discontinuous progress around the development of AGI

#11

Post by mrfusion » Sun, Feb 25, 2018, 1:58 PM UTC

nopinsight wrote: post #5

We know that for most problems, a group of people tends to be better at problem solving than an individual [1]. Even if AI technology only reaches human-level and does not exceed it, continually increasing efficiency would make AI smarter than any small groups of humans and immense bandwidth relative to human communications would make it more effective than any large human organizations. [1] https://aiimpacts.org/coo…

That counterargument makes a lot of sense. I wonder why I never see it discussed anywhere?

It might explain why humans are all close in intelligence to each other. Eg even with seven billion people we’ve never seen someone with a 500 iq.

Or that you can explain the work done by the smartest human to someone with average iq if you invested some time building up their knowledge.

Re: Likelihood of discontinuous progress around the development of AGI

#12

Post by nopinsight » Sun, Feb 25, 2018, 2:21 PM UTC

mrfusion wrote: post #11

nopinsight wrote: post #5

We know that for most problems, a group of people tends to be better at problem solving than an individual [1]. Even if AI technology only reaches human-level and does not exceed it, continually increasing efficiency would make AI smarter than any small groups of humans and immense bandwidth relative to human communications would make it more effective than any large human organizations. [1] https://aiimpacts.org/coo…

That counterargument makes a lot of sense. I wonder why I never see it discussed anywhere? It might explain why humans are all close in intelligence to each other. Eg even with seven billion people we’ve never seen someone with a 500 iq. Or that you can explain the work done by the smartest human to someone with average iq if you invested some time building up their knowledge.

Thanks. I came up with that myself although I would like to see if someone can poke holes on it [1]. I am aware that many NP-hard problems, for example, usually have good approximate solutions in non-pathological cases. So that could be a big hole in that argument.

Could a theoretical computer scientist give more insight on this? In particular, what is the hardness of approximate solutions to most problems that could have strong impact in the real world?

[1] I did, in passing, find a similar idea discussed somewhere afterwards. If someone has a reference for that, I would be glad to have it.

Re: Likelihood of discontinuous progress around the development of AGI

#13

Post by sampo » Sun, Feb 25, 2018, 2:26 PM UTC

darawk wrote: post #4

> This argument seems weak to us currently, but further research could resolve these questions in directions that would make it compelling: > Are individual humans radically superior to apes on particular measures of cognitive ability? What are those measures, and how plausible is it that evolution was (perhaps indirectly) optimizing for them? Yes, clearly. All the measures defined in this article are arbitrary (e.g.…

> let's propose an arbitrary one for intelligence: Ability to prove mathematical theorems

The number of mathematical theorems proven by humankind has progressed in a continuous manner (as much as a quantity measured by integer numbers can) during history.

Re: Likelihood of discontinuous progress around the development of AGI

#14

Post by simonh » Sun, Feb 25, 2018, 2:52 PM UTC

sampo wrote: post #13

darawk wrote: post #4

> This argument seems weak to us currently, but further research could resolve these questions in directions that would make it compelling: > Are individual humans radically superior to apes on particular measures of cognitive ability? What are those measures, and how plausible is it that evolution was (perhaps indirectly) optimizing for them? Yes, clearly. All the measures defined in this article are arbitrary (e.g.…

> let's propose an arbitrary one for intelligence: Ability to prove mathematical theorems The number of mathematical theorems proven by humankind has progressed in a continuous manner (as much as a quantity measured by integer numbers can) during history.

Despite us starting with the simplest theorems and progressing to significantly harder and harder ones over time? Wow. That’s actually very impressive.

Re: Likelihood of discontinuous progress around the development of AGI

#15

Post by jwellt » Sun, Feb 25, 2018, 3:08 PM UTC

dane-pgp wrote: post #10

It's an interesting read, but perhaps it only succeeds in pointing out the gaps in our knowledge. When I read their counterargument to the possibility of intelligence explosion: "Positive feedback loops are common in the world, and very rarely move fast enough and far enough to become a dominant dynamic in the world." the idea that immediately comes to mind is the Harmless Supernova Fallacy, described on this (obnoxi…

I think the reasoning in the arguments is left a little loose on purpose simply because it is so difficult make strong arguments about something we know so little about.

Re: Likelihood of discontinuous progress around the development of AGI

#16

Post by paganel » Sun, Feb 25, 2018, 3:18 PM UTC

sampo wrote: post #13

darawk wrote: post #4

> This argument seems weak to us currently, but further research could resolve these questions in directions that would make it compelling: > Are individual humans radically superior to apes on particular measures of cognitive ability? What are those measures, and how plausible is it that evolution was (perhaps indirectly) optimizing for them? Yes, clearly. All the measures defined in this article are arbitrary (e.g.…

> let's propose an arbitrary one for intelligence: Ability to prove mathematical theorems The number of mathematical theorems proven by humankind has progressed in a continuous manner (as much as a quantity measured by integer numbers can) during history.

That only says something about maths, not about humans. For comparison, you could say that the number of literary masterpieces has been increased at what looks to be random intervals for the last 2 or 3 millennia, with no continuous progress in sight (i.e. the Spanish language has not had a new Cervantes for 400 years, the same applies to Skakespeare and the English language or to Dante and Italian). But this being an website mostly addressed to people who focus on technical stuff I expect a reply like “literature doesn’t count, it’s just words”.

Re: Likelihood of discontinuous progress around the development of AGI

#17

Post by the8472 » Sun, Feb 25, 2018, 3:24 PM UTC

mrfusion wrote: post #11

nopinsight wrote: post #5

We know that for most problems, a group of people tends to be better at problem solving than an individual [1]. Even if AI technology only reaches human-level and does not exceed it, continually increasing efficiency would make AI smarter than any small groups of humans and immense bandwidth relative to human communications would make it more effective than any large human organizations. [1] https://aiimpacts.org/coo…

That counterargument makes a lot of sense. I wonder why I never see it discussed anywhere? It might explain why humans are all close in intelligence to each other. Eg even with seven billion people we’ve never seen someone with a 500 iq. Or that you can explain the work done by the smartest human to someone with average iq if you invested some time building up their knowledge.

> Eg even with seven billion people we’ve never seen someone with a 500 iq.

For two reasons. First of all IQ is defined as a normal distribution with a mean of 100 and a sd15. So by definition only 0.1% of the population can have an IQ of above 145 and this fraction is rapidly diminishing with additional standard deviations.

Secondl: the definition matches reality reasonably well because intelligence is a polygenetic trait and many small factors add up to a normal distribution.[0] It is statistically unlikely that natural processes will yield an IQ500 human. You would have to engineer one. Which leads us back to the AGI concern.

[0] https://en.wikipedia.org/wiki/Central_limit_theorem

Re: Likelihood of discontinuous progress around the development of AGI

#18

Post by coolaliasbro » Sun, Feb 25, 2018, 3:37 PM UTC

Retric wrote: post #6

nopinsight wrote: post #5

We know that for most problems, a group of people tends to be better at problem solving than an individual [1]. Even if AI technology only reaches human-level and does not exceed it, continually increasing efficiency would make AI smarter than any small groups of humans and immense bandwidth relative to human communications would make it more effective than any large human organizations. [1] https://aiimpacts.org/coo…

This also assumes intelligence is the limiting factor for solving many problems. I suspect information and computation is probably the larger factor for most major issues. The smartest player possible would still lose poker to someone that can read their hand.

> The smartest player possible would still lose poker to someone that can read their hand.

And let's not forget those cases where the smartest player is the one that can read your hand.

https://en.m.wikipedia.org/wiki/Stu_Ungar

Re: Likelihood of discontinuous progress around the development of AGI

#19

Post by s1dechnl » Sun, Feb 25, 2018, 3:38 PM UTC

nopinsight wrote: post #5

We know that for most problems, a group of people tends to be better at problem solving than an individual [1]. Even if AI technology only reaches human-level and does not exceed it, continually increasing efficiency would make AI smarter than any small groups of humans and immense bandwidth relative to human communications would make it more effective than any large human organizations. [1] https://aiimpacts.org/coo…

[AGI Developer here]

> the real barrier for AGI is software and not hardware.

This is indeed true. It has actually been true for some time. AGI is computable on present day hardware if one has enough knowledge on how to correctly structure it. A fundamental understanding of intelligence is the first step. How one crafts this understanding into software is the second step. Hardware reached capability in recent years.

> exploding intelligence

There is no such thing. Time is still required like with all things. Teaching/learning/interaction is still required. Furthermore, A controller/overseer of the system can more than adequately limit progress they are not comfortable with. I find the idea of exploding intelligence/overnight super AI to be pure fantasy not at all aware as to the structure of AGI.

> If an organization can marginally predict stock price movements better than the rest of the world..

The problem is this kind of thinking... AGI is achieved and people rush to apply it to games to get rich. Sorry, that will not occur. It will not occur because the stock market is fundamentally a [game]. A game with disadvantaged players. A game with incomplete information. A game whose rules/dynamics change frequently to suit inside players. One could make all of the accurate predictions they wanted, if the game changes underneath you or before you can act, your lofty predictions have no real world value and that's exactly how the market behaves.

Re: Likelihood of discontinuous progress around the development of AGI

#20

Post by coolaliasbro » Sun, Feb 25, 2018, 3:43 PM UTC

nopinsight wrote: post #9

Retric wrote: post #6

Earlier quoted context omitted.

This also assumes intelligence is the limiting factor for solving many problems. I suspect information and computation is probably the larger factor for most major issues. The smartest player possible would still lose poker to someone that can read their hand.

I agree that information is crucial to achieve a 'win' for many goals. Given today's amount of information on the Internet as well as electronic money and access to most officials, barring some sort of inviolable built-in moral core, an AGI would be able to use any methods, overt and covert, direct and cunning, technical and social, to achieve its information goals. [1] Since an AGI can copy itself and be available a…

> I think an AGI will possibly be capable of changing its own core, so a much more reliable safeguard is to make sure that it does not want to change it.

Assuming an AGI's consciousness would be anything like a human's, changes to its moral core might be largely dictated by environment, in which case it would be a good idea to be pals.