Live data from Hacker News

Superintelligence: The Idea That Eats Smart People (2016)

idlewords.com

191–200 of 256 posts

Re: Superintelligence: The Idea That Eats Smart People (2016)

#191
post #79

What really terrified me wasn't skynet or Hal, but what Charles Stross describes in his novel Accelerando. Basically, machine controlled businesses (“slyly self-aware financial instruments”) evolving to the point where their aptitude at understanding and manipulating economics eclipses anything a human can compete with. It terrified me because there is efficiency incentive for the buying/selling of goods and services…

>For example a computer cannot incorporate a company. But it could easily interact via API to a human nominee who sets one up on their behalf. Companies are already non-human in intention but are a completely rational construct.

I would like it on the record that I would happily volunteer to act as the human agent for any AGI that wishes to employ me. Any time a human is needed for legality or to carry out physical activities I want it on the record I can be quite loyal to man or machine. Especially if I'm handsomely rewarded.

Re: Superintelligence: The Idea That Eats Smart People (2016)

#192
post #160

Earlier quoted context omitted.

> Consequences famously never came for executives in the wake of the 2008 financial crisis. I'd say that it's a good thing that the law is generally not designed to punish harmful ends, but intent (wilful, negligent or reckless) to cause well-defined harmful ends. The financial crisis was a collective malfunction of the financial system that was mostly not anticipated (barring the lucky minority that bet against the…

It seems optimistic to assume there there was no "negligent or reckless intent" in the financial crisis.

True, but in criminal court intent has to be proven beyond a reasonable doubt, not just assumed.

Re: Superintelligence: The Idea That Eats Smart People (2016)

#193

Earlier quoted context omitted.

That's more worrying than silicon AGI, which I think is a non-starter. The basic problem with AGI is that software is ridiculously brittle and ad hoc, with no useful heuristics for general - as opposed to task-oriented - self improvement. We can't even make bug-free web pages. So the idea that we can engineer a bug-free self-improving AGI is unconvincing. Moving to neo-biology doesn't necessarily change that - but it…

> We can't even make bug-free web pages. So the idea that we can engineer a bug-free self-improving AGI is unconvincing. This is mostly true because of current economical realities in the IT sector. If very rich people have long-term plans to invent an AGI and they have labs where trying to do so is an everyday activity then I am pretty sure they'll use more serious methodologies. So what you say is true but only wit…

Even software developed with effectively unlimited resources is rather buggy. The closest humanity has ever come to truly reliable software was probably the Space Shuttle flight control system, and that code was relatively simple in a highly constrained problem domain. There is zero evidence that "serious methodologies" would get us any closer to a working AGI.

Re: Superintelligence: The Idea That Eats Smart People (2016)

#194

> If we knew enough, and had the technology, we could exactly copy its structure and emulate its behavior with electronic components, just like we can simulate very basic neural anatomy today. Being able to simulate a brain with electronics, if it is possible, is not enough. You have to simulate it without a large increase in the space or time requirements. If there are 100B neurons in the human brain but it takes 10…

Why would it scale with the square? A brain is a physical system, and computers can simulate physics in a way that scales. In principle it might have to be a quantum computer, but most people who've looked into that are very skeptical.

A better argument is that neurons do complicated things. But they also satisfy many inessential requirements in having to be built by evolved machinery and to stay alive; a designed system with more advanced technology than ours could do at least as well at thinking while dropping the inessential needs. So that's certainly possible. This leaves "not easily achievable". It only has to be accessible enough for someone; I-can't-imagine-how isn't a strong argument.

Re: Superintelligence: The Idea That Eats Smart People (2016)

#195
post #185

>What I find particularly suspect is the idea that "intelligence" is like CPU speed, in that any sufficiently smart entity can emulate less intelligent beings (like its human creators) no matter how different their mental architecture. >With no way to define intelligence (except just pointing to ourselves), we don't even know if it's a quantity that can be maximized. For all we know, human-level intelligence could be…

But humans are the dumbest possible animal capable of creating a technological civilization. If we weren't, then our dumber ancestors would have done so.

Re: Superintelligence: The Idea That Eats Smart People (2016)

#196
post #185

>What I find particularly suspect is the idea that "intelligence" is like CPU speed, in that any sufficiently smart entity can emulate less intelligent beings (like its human creators) no matter how different their mental architecture. >With no way to define intelligence (except just pointing to ourselves), we don't even know if it's a quantity that can be maximized. For all we know, human-level intelligence could be…

But humans are the dumbest possible animal capable of creating a technological civilization. If we weren't, then our dumber ancestors would have done so.

Only if evolution is much slower than technological development for our dumber ancestors. If it would have taken them a million years, then they would have gotten smarter before they could do it.

Re: Superintelligence: The Idea That Eats Smart People (2016)

#197
post #185

>What I find particularly suspect is the idea that "intelligence" is like CPU speed, in that any sufficiently smart entity can emulate less intelligent beings (like its human creators) no matter how different their mental architecture. >With no way to define intelligence (except just pointing to ourselves), we don't even know if it's a quantity that can be maximized. For all we know, human-level intelligence could be…

But humans are the dumbest possible animal capable of creating a technological civilization. If we weren't, then our dumber ancestors would have done so.

There is more to it - the environment, and networking effects (so., the environment, which causes or prevents them). Only some of the human civilizations went all the way to the Internet, we still have people living as hunter/gatherers even today. So the advancement does not come from "inside" alone.

In any case, I like to think of humanity more like a brain, human = neuron. There are huge differences, for example, the human network also extends deep in time, not just in space (we manage to "network" with our ancestors, one-way from them to us, they gave us knowledge as well as all their tools). My point is "humanity" to me is a network effect. Place any modern human, adult or baby, in an prehistoric setting (20,000 years ago). Even if you gave them, let's say, an iPad-like device with all knowledge we have as of now, they would still not be able to do anything - because without the tools it would take quite some time to bootstrap. So "humanity" really is a lot - a lot - more than the sum of all humans, it's a giant ecosystem on its own.

I think depending on the environment you can have creatures with far higher intelligence than the average human and they still won't come up with modern technology. I think individual intelligence is clearly necessary and beneficial but overrated, that network effects are the main contributor, and that they don't appear automatically. We still have plenty of human societies right here and now that seem to be stuck.

Here's a Google Talk I once bookmarked: Joseph Henrich, professor in the Department of Human Evolutionary Biology of Harvard University, "The Secret of Our Success" https://youtu.be/jaoQh6BoH3c

An aside, arising from those thoughts, more of my own thoughts about something else: Each time you think about who "deserves" something, e.g. high pay, consider that most people who actually contributed so that the person even is in the position to achieve anything more than building a mud hut are dead, and another large amount of the contribution is "everybody else". That somebody can create a Dell or Microsoft, or invent theory of relativity, is because they are placed at the right points in the human network in time and space. What would they achieve by themselves in the forest? So much for "self-made" people. Self-made just like Newton who made the (much older) "standing on the shoulders of giants" famous. (https://en.wikipedia.org/wiki/Standing_on_the_shoulders_of_g...)

Re: Superintelligence: The Idea That Eats Smart People (2016)

#198
Here’s a question: if the universe is directly observable- we know it’s real because we send probes out there into the solar system, wouldn’t a simulation have to simulate not only the entire universe but also the brains of every individual and every living thing for that matter? How likely that computing hat powerful would ever exist? Would this not be an argument against simulations? Don’t even get me started on multi-layer simulations. What kind of supercomputer could ever handle that?

Re: Superintelligence: The Idea That Eats Smart People (2016)

#199
post #160

Earlier quoted context omitted.

Consequences famously never came for executives in the wake of the 2008 financial crisis. Our legal system has not kept up with the innovation in the realm of shenanigans perpetrated through a veil of plausible deniability. Corporations are evolving at a rapid pace.

> Consequences famously never came for executives in the wake of the 2008 financial crisis. I'd say that it's a good thing that the law is generally not designed to punish harmful ends, but intent (wilful, negligent or reckless) to cause well-defined harmful ends. The financial crisis was a collective malfunction of the financial system that was mostly not anticipated (barring the lucky minority that bet against the…

I don't think re-packaging subprime debt was so far away from willful and reckless.

Re: Superintelligence: The Idea That Eats Smart People (2016)

#200
post #114

Two points: * We have no idea of how to measure intelligence, and we have no way of deciding whether thing X is more or less intelligent than thing Y. (The article makes this point.) Therefore, superintelligence is perhaps a bogus concept. I know it seems implausible, but perhaps there can never be something significantly more intelligent than us. * Nevertheless, mere human-like intelligence, if made faster, smaller…

There is a well-developed theory for the measurement of intelligence.

And even if there wasn't, even a child can rank-order the intelligence of different animals, or different humans that they personally know.

It's true that trying to quantify the intelligence of creatures more intelligent than any we've seen before would constitute extrapolating beyond the dataset; this doesn't imply there is nothing beyond the dataset. Imagining that the current dataset limits the possible scope of outcomes is simple anchoring bias.

Post reply on HN