Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The impossibility of intelligence explosion

medium.com

Post reply on HN
Print view View original

91–100 of 143 posts

Re: The impossibility of intelligence explosion

#91

Post by breuleux »

michaelmrose wrote: post #59
breuleux wrote: post #36

Earlier quoted context omitted.

> We can not simply look in our brains/minds and read our source code, nor easily tweak it to see what would happen without great risks to our lives. A computer could. That is not a given. The neural network type of AI, which is so far the most promising avenue, is about as opaque as a human brain. It is far from obvious that even a superintelligent AI could understand what its own brain does, let alone modify it in…

The ability to make a billion identical somethings would seem to be more valuable than any possible improvement you could make. The other trade offs you are imagining seem to be worse than any possible gain.

Straight copying is not a robust way to scale: any vulnerability found in a single copy will be found in all of them, so you end up with a very brittle system. As far as copying goes, you want a smarter scheme than that, where e.g. only truly useful information is transferred and the rest is rebuilt around that. The breadth of easy information access current computers allow is good for the really naive approaches we currently use, but probably overkill relative to more efficient schemes.

Re: The impossibility of intelligence explosion

#92

Post by Bartweiss »

akvadrako wrote: post #76
Bartweiss wrote: post #71

Earlier quoted context omitted.

Probably you don't, it's likely an unverifiable proposition. But "better at ethics" is a completely different question than "better at implementing a given ethical stance". The people frightened of intelligence explosion are worried about something like an AI version of existentialism: a mind that accepts some moral system without even trying to justify it, and then optimizes accordingly. It's certainly possible to j…

> I've seen lots of essays (not this one) claim that morality will "inherently" emerge from intelligence, which I think is absurd. Shit, my moral views aren't an 'inherent' product of anything except my evolution-shaped brain that feels empathy. You're evolution-shaped brain is what an AI will become too. It won't start out super-human smart; it will evolve to an equal level. Then keep going. Morals are basically low…

...huh?

AI will reach parity with humans before it exceeds them, agreed. But intelligence parity is not the same as "matching my evolution-shaped brain". Feeling empathy for other humans is not a requirement for intelligence; I routinely make supposedly-irrational choices to help humans in non-reciprocal settings because I feel empathy. There's no reason to think an AI that can write code as well as me will feel the same.

> Morals are basically lower-level approximations which are beneficial for survival and AI's will certainly evolve those too.

AI will evolve approximations which are beneficial for survival. We agree there. But why would a bunch of approximations that helped my ancestors survive in low-tech communal environments benefit a strong AI trained with high tech, minimal survival needs, and no "peer group"?

I fully expect strong AI will pursue some set of real-world goals which aren't justified except by the anthropic principle. But the assumption that those goals will match human goals seems to completely ignore the fact that the first strong AI will live in a vastly different environment than the first human.

Re: The impossibility of intelligence explosion

#93

Post by lodi »

nmeofthestate wrote: post #32

"We understand flight - we can observe birds in nature, to see how flight works. The notion that aircraft capable of supersonic speeds are possible is fanciful."

My thoughts exactly. The author is concluding that since we've never seen supersonic wing-flapping in nature, we'll never see supersonic flight. Not only is that an unwarranted implication, in a mathematical sense, but it also doesn't cover the possibility of inventing something fundamentally different to what we've observed in the universe thus far.

Using human and octopus examples as proof of anything is ignoring that general AI will be fundamentally different to anything that we've observed on earth, and therefore invalidates any attempt to extrapolate from history. There has never been an intelligence that could copy/paste itself intact with its existing memories. There has never been an intelligence that could literally span the whole world with a single consciousness. And so on.

> There is no such thing as “general” intelligence. The intelligence of a human is specialized in the problem of being human.

That's exactly what people are worried about. When the AI specializes itself to be better at being a human than actual humans. It doesn't matter if it's totally general and can be applied to any problem whatsoever; human intelligence is also just "general enough".

> Will the superhuman AIs of the future, developed collectively over centuries, have the capability to develop AI greater than themselves? No, no more than any of us can.

Okay, so instead of single AI's developing next-gen AI's, it'll be "AI civilization" developing next-gen AI's. This still poses exactly the same existential risk to humanity.

> We didn’t make greater progress in physics over the 1950–2000 period than we did over 1900–1950 — we did, arguably, about as well.

Well I certainly know we made exponentially more progress in those years than from the years 900-950? Nevermind say 100,000-5,000 BCE.

Re: The impossibility of intelligence explosion

#94

Post by naasking »

10-6 wrote: post #11

The author makes several points about AI/intelligence and recursive systems that I think a lot of people who think "AI will take over the world and replace" humans [1][2][3] don't understand. He argues that general intelligence by itself is not really something that exists, our brains exist within a broader system (environment, culture, bodies, etc.) which is something you can read more about in embodied cognition: h…

> Automation of jobs and tasks is very real, but completely replacing humans and potentially destroying us all is a joke and only the people who no nothing about AI/brains think this.

Don't be absurd. Brains are simply biochemical computers. The idea of electrical or other types of computers superceding us is perfectly realizable. The only real debate is over the timeline.

Re: The impossibility of intelligence explosion

#95

Post by chillee »

lodi wrote: post #90
Retric wrote: post #63

Earlier quoted context omitted.

Games have a clear ends goal. How to you measure getting better at ethics?

Maximize human population divided by the time integral of all human suffering (taken from now to the heat death of the universe.)

This kind of stuff is pretty tricky. If you only account for average human suffering, not only do you not account for happiness but you fall into the trap of concluding that it's best to kill all people suffering.

Re: The impossibility of intelligence explosion

#96

Post by Strilanc »

lodi wrote: post #90
Retric wrote: post #63

Earlier quoted context omitted.

Games have a clear ends goal. How to you measure getting better at ethics?

Maximize human population divided by the time integral of all human suffering (taken from now to the heat death of the universe.)

Congratulations, your AI wants to convert all matter in the universe into lobotomized humans.

Re: The impossibility of intelligence explosion

#97

Post by jessaustin »

AstralStorm wrote: post #88
munificent wrote: post #85

Earlier quoted context omitted.

> What specialized intelligence, then, is human intelligence? I think the author just means "the skills needed to pilot a human body on Earth in a normal human social environment". I see no tautology, and the preceding sentence about octopuses makes the author's meaning here pretty clear.

I suppose one has to know calculus and topology to navigate Earth. Worth a smattering of quantum physics and wave physics. Oh wait. Those do not fall under such narrow definition of intelligence.

How many insects or birds know "calculus and topology"? You're straining so much here, to be both narrow and broad, that you contradict yourself.

Re: The impossibility of intelligence explosion

#98

Post by spuz »

naasking wrote: post #94
10-6 wrote: post #11

The author makes several points about AI/intelligence and recursive systems that I think a lot of people who think "AI will take over the world and replace" humans [1][2][3] don't understand. He argues that general intelligence by itself is not really something that exists, our brains exist within a broader system (environment, culture, bodies, etc.) which is something you can read more about in embodied cognition: h…

> Automation of jobs and tasks is very real, but completely replacing humans and potentially destroying us all is a joke and only the people who no nothing about AI/brains think this. Don't be absurd. Brains are simply biochemical computers. The idea of electrical or other types of computers superceding us is perfectly realizable. The only real debate is over the timeline.

It's not absurd to believe that humans will never be able to invent machines that match human level intelligence. Firstly, it's not necessarily logical to assume that brains are simply computers because we've never created or even imagined a computer that works anything like a brain. Even if it's possible, it might be beyond human intelligence to invent such a computer.

I like to think of AGI like the Warp Drive. It might well be possible, but we might never develop the capability to make one.

Re: The impossibility of intelligence explosion

#99

Post by amag »

naasking wrote: post #94
10-6 wrote: post #11

The author makes several points about AI/intelligence and recursive systems that I think a lot of people who think "AI will take over the world and replace" humans [1][2][3] don't understand. He argues that general intelligence by itself is not really something that exists, our brains exist within a broader system (environment, culture, bodies, etc.) which is something you can read more about in embodied cognition: h…

> Automation of jobs and tasks is very real, but completely replacing humans and potentially destroying us all is a joke and only the people who no nothing about AI/brains think this. Don't be absurd. Brains are simply biochemical computers. The idea of electrical or other types of computers superceding us is perfectly realizable. The only real debate is over the timeline.

Superseding us in what way? The machine is already better at chess and Go, clearly not at classifying images[1], but the issue here is that the author argues that there is no such thing as a general intelligence. It's all situational.

On a separate note, does it even matter? Some people argue it's important so that we can put AI safe-guards in place, to protect mankind. But either:

A) Intelligence explosion won't happen and those safe-guards are not needed.

B) Intelligence explosion happens and the supersmart AI will easily find its way around feeble safe-guards put in place by the inferior humans.

While the end results are not the same; in either case the safe-guards are useless, though Asimov sure made a good story about them.

[1] https://arxiv.org/abs/1710.08864

Re: The impossibility of intelligence explosion

#100

Post by jessaustin »

titzer wrote: post #21

> A high-potential human 10,000 years ago would have been raised in a low-complexity environment, likely speaking a single language with fewer than 5,000 words, would never have been taught to read or write, would have been exposed to a limited amount of knowledge and to few cognitive challenges. The situation is a bit better for most contemporary humans, but there is no indication that our environmental opportunitie…

If we imagine an Einstein of the 23rd century, whose great discoveries will enable humanity to cross the universe in a moment while expending the energies of many stars, and then we imagine that person educated in a USA public school in 2017, can we imagine her making those same discoveries? Of course not, which is most of what TFA is saying here. Perhaps "general AI" will increase the rate of improvement, but even the most gifted AI will be limited by the context in which it operates.
Post reply on HN