Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Superintelligence: The Idea That Eats Smart People (2016)

idlewords.com

Post reply on HN
Print view View original

181–190 of 256 posts

Re: Superintelligence: The Idea That Eats Smart People (2016)

#181

Post by notahacker »

atq2119 wrote: post #158
lsc wrote: post #83

Earlier quoted context omitted.

Eh, if you are talking about something truly human-like rather than what people usually mean when they say AI these days, I think there are other limits. We're running up against limits on feature-size in our integrated circuits. If you invent an AGI that can make smarter copies of itself... it's pretty likely those same physical limits will continue to apply, meaning that each iteration will be better than the last,…

I think you need to distinguish between AGI and the singularity. We know that nothing in physics prevents AGI, because humans exist. And AGI itself, even if it isn't entirely runaway, could pose a challenge. (I'd argue that corporations are a form of AGI that is already a major challenge for our societies.) The singularity as described by its acolytes, on the other hand, runs into the kind of physical limits you desc…

If we take humans as our model for AGI, it's worth noting that well-established networks of humans have proved efficient at ahem... deleting more-intelligent but potentially threatening human minds, that status and EQ has tended to prevail over IQ when it comes to manipulating humans into acting against their interests, that relatively close physical copies of human minds often have very dissimilar goals and abilities so self-replicating AGIs might ultimately be self-defeating... but above all that we've already got multiple threats to humanity in the form of human intelligences, augmented by technology that's often as dumb as it is deadly.

Re: Superintelligence: The Idea That Eats Smart People (2016)

#182

Post by winchling »

ian0 wrote: post #79

What really terrified me wasn't skynet or Hal, but what Charles Stross describes in his novel Accelerando. Basically, machine controlled businesses (“slyly self-aware financial instruments”) evolving to the point where their aptitude at understanding and manipulating economics eclipses anything a human can compete with. It terrified me because there is efficiency incentive for the buying/selling of goods and services…

But we already have a situation where smart but nasty people can set up companies. Either they are rent-seekers in which case the harm they can do is limited, or they're creative, in which case their activity benefits society.

Even what they do with their money and influence in their spare time is limited in its harmful effects, because as personalities they are at root afraid of other people and/or inflicted by a profound sense of lack. These qualities ultimately lead to dumb decisions.

To summarise: buying and selling are ultimately pretty boring. So is controlling people. Real power is the power to create, which is fun.

Re: Superintelligence: The Idea That Eats Smart People (2016)

#183

Post by grkvlt »

crunchyfrog wrote: post #149

> If Einstein tried to get a cat in a carrier, and the cat didn't want to go, you know what would happen to Einstein. > > He would have to resort to a brute-force solution that has nothing to do with intelligence, and in that matchup the cat could do pretty well for itself. Wouldn't it be better to simply motivate the cat to get in the carrier with a can of tuna fish? This is a very simple form of modeling the cat's…

> Wouldn't it be better to simply motivate the cat to get in the carrier with a can of tuna fish?

One does not simply motivate a cat with a can of tuna fish. I mean, have you even owned a cat? If the little bastard doesn't want to get in the cat carrier, no amount of delicious tuna will encourage it, and you're going to end up using brute force and getting mauled...

Re: Superintelligence: The Idea That Eats Smart People (2016)

#184

Post by jessriedel »

jessriedel wrote: post #179

Under "The Argument From Complex Motivations", I encourage you to read the linked paper on the orthogonality thesis https://www.fhi.ox.ac.uk/wp-content/uploads/Orthogonality_An... and then compare it to the OP discussion. The OP doesn't address any of the arguments in the paper. His response is just "nah".

For instance, go to the bottom of page 8 and see this:

> Thus to deny the Orthogonality thesis is to assert that there is a goal system G, such that, among other things:

> 1. There cannot exist any efficient real-world algorithm with goal G.

> 2. If a being with arbitrarily high resources, intelligence, time and goal G, were to try design an efficient real-world algorithm with the same goal, it must fail.

> ...4. If a high-resource human society were highly motivated to achieve the goal G, then it could not do so (here the human society itself is seen as the algorithm).

> ...6. There cannot exist any pattern of reinforcement learning that would train a highly efficient real-world intelligence to follow the goal G.

> All of these seem extraordinarily strong claims to make!

The OP doesn't consider any of these.

Re: Superintelligence: The Idea That Eats Smart People (2016)

#185

Post by Isamu »

>What I find particularly suspect is the idea that "intelligence" is like CPU speed, in that any sufficiently smart entity can emulate less intelligent beings (like its human creators) no matter how different their mental architecture.

>With no way to define intelligence (except just pointing to ourselves), we don't even know if it's a quantity that can be maximized. For all we know, human-level intelligence could be a tradeoff. Maybe any entity significantly smarter than a human being would be crippled by existential despair, or spend all its time in Buddha-like contemplation.

Glad to see this articulated here, this is one of the many problems I have with this topic.

Re: Superintelligence: The Idea That Eats Smart People (2016)

#186

Post by wzdd »

jessriedel wrote: post #184
jessriedel wrote: post #179

Under "The Argument From Complex Motivations", I encourage you to read the linked paper on the orthogonality thesis https://www.fhi.ox.ac.uk/wp-content/uploads/Orthogonality_An... and then compare it to the OP discussion. The OP doesn't address any of the arguments in the paper. His response is just "nah".

For instance, go to the bottom of page 8 and see this: > Thus to deny the Orthogonality thesis is to assert that there is a goal system G, such that, among other things: > 1. There cannot exist any efficient real-world algorithm with goal G. > 2. If a being with arbitrarily high resources, intelligence, time and goal G, were to try design an efficient real-world algorithm with the same goal, it must fail. > ...4. If…

The OP's point is:

1. The "superintelligence" crowd are capable of making very convincing arguments which he simply cannot refute on their own terms (I guess he'd say your example is one of them)

2. Since he can't refute them, if he engages with them, he will probably end up accepting them

3. However other groups exist which a) present equally-convincing (to the OP) arguments; and b) believe in a future which contradicts that espoused by the superintelligence crowd

4. With multiple sets of equally-convincing arguments leading to contradictory outcomes, it is not rational to accept any one set.

This is not a conclusion which sits well with tech people, because they like to assume they are capable of engaging with each set of arguments and deciding on the most likely set. OP's point is that because each group is really good at making these arguments, it's easy to get this decision-making process wrong. I think that's a pretty great point, but it requires accepting the fact that you may be less than perfect at picking apart 'superingelligence' arguments and premises.

Re: Superintelligence: The Idea That Eats Smart People (2016)

#187

Post by SpicyLemonZest »

ian0 wrote: post #79

What really terrified me wasn't skynet or Hal, but what Charles Stross describes in his novel Accelerando. Basically, machine controlled businesses (“slyly self-aware financial instruments”) evolving to the point where their aptitude at understanding and manipulating economics eclipses anything a human can compete with. It terrified me because there is efficiency incentive for the buying/selling of goods and services…

>Basically, machine controlled businesses (“slyly self-aware financial instruments”) evolving to the point where their aptitude at understanding and manipulating economics eclipses anything a human can compete with.

I'd argue this is already true to a substantial degree. When Lehman Brothers collapsed, for example, the world's brightest economic minds came together and still couldn't fully contain the fallout.

Re: Superintelligence: The Idea That Eats Smart People (2016)

#188

Post by PhasmaFelis »

gnode wrote: post #160
rectang wrote: post #155

Earlier quoted context omitted.

Consequences famously never came for executives in the wake of the 2008 financial crisis. Our legal system has not kept up with the innovation in the realm of shenanigans perpetrated through a veil of plausible deniability. Corporations are evolving at a rapid pace.

> Consequences famously never came for executives in the wake of the 2008 financial crisis. I'd say that it's a good thing that the law is generally not designed to punish harmful ends, but intent (wilful, negligent or reckless) to cause well-defined harmful ends. The financial crisis was a collective malfunction of the financial system that was mostly not anticipated (barring the lucky minority that bet against the…

It seems optimistic to assume there there was no "negligent or reckless intent" in the financial crisis.

Re: Superintelligence: The Idea That Eats Smart People (2016)

#189

Post by abecedarius »

lsc wrote: post #83
harshreality wrote: post #74

Earlier quoted context omitted.

You're essentially arguing that AGI can't exist, and you're assuming when people say "AI will turn the world to paperclips" they mean a really smart go-playing neural net could do that. Of course it can't. Everyone agrees there is some magic smoke necessary for AGI that we haven't figured out yet. The arguments that AI will eat the world are about AGI , not current instances of narrow AI. It may be true that an AGI w…

Eh, if you are talking about something truly human-like rather than what people usually mean when they say AI these days, I think there are other limits. We're running up against limits on feature-size in our integrated circuits. If you invent an AGI that can make smarter copies of itself... it's pretty likely those same physical limits will continue to apply, meaning that each iteration will be better than the last,…

Integrated circuits are nowhere near what's possible. Brains are possible. Brains send signals at roughly the speed of sound. Copper wires send signals at roughly the speed of light. It's unlikely something much like a brain couldn't use copper wires, it's just too hard for us to build yet.
Post reply on HN