Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Superintelligence: The Idea That Eats Smart People (2016)

idlewords.com

Post reply on HN
Print view View original

61–70 of 253 posts

Re: Superintelligence: The Idea That Eats Smart People (2016)

#61

Post by ACCount37 »

jandrese wrote: post #30
blamestross wrote: post #6

AI Superintelligence doesn't scare me for the same reasons "grey goo" doesn't scare me. We are awash in self-replicating machines. The biosphere is already a grey-goo apocalypse. Any new competitors have a serious moat to cross to out compete any existing self-replicators. We are awash in intelligent agents. Our society (and meta society) is full of superhuman agents already. There is a huge moat for any new intellig…

I have the same feeling. I'm not worried about superintelligent AI because we are only training them on human level intelligence. By what mechanism does our current AI technology take the leap to technologies that humans have never conceived of? Our current AI is more like a fancy Google search than some kind of machine God.

"Human level intelligence" is not some sort of hard ceiling. We already can create AIs that are vastly superhuman in narrow domains. It would be the height of hubris to claim that a broadly superhuman AI is impossible - that would require human brain to be the pinnacle of general intelligence.

How do we get to ASI? That's what recursive self-improvement is about.

If AGI is reachable, then we can make AI that, in turn, makes improved successor AIs. The performance goes up. It's not bounded by human intelligence - it's bounded by how much the previous generation of AI could improve upon itself.

We don't have a stable recipe for RSI yet, but AI development is already AI-assisted. It's just that the "improvement" loops of today are long, and require plenty of human input. Betting against RSI is betting that it'll stay that way forever - that tightening the loop and removing humans from it is fundamentally impossible.

Re: Superintelligence: The Idea That Eats Smart People (2016)

#62

Post by [deleted] »

somesortofthing wrote: post #59
ctoth wrote: post #54

Earlier quoted context omitted.

I keep seeing this and I want to speak to it. When Dario and others say things like "this is happening and we should probably figure out what to do about it" what ends up happening is people hear "this is happening," see that the person warning them is the person doing the thing, and then short-circuit. "Why can't you just stop then?" Dario's point, and the point of the people actually trying to solve the problem, is…

A US-China AGI ban treaty could prevent superintelligence indefinitely. Data centers are hard to hide. Have fun buying GPUs when you're cut off from all global payments. America would have to make some unpleasant concessions but that seems like a solid trade for preventing a wide variety of nightmare futures.

[deleted]

Re: Superintelligence: The Idea That Eats Smart People (2016)

#63

Post by mbgerring »

ctoth wrote: post #54
cute_boi wrote: post #12

Earlier quoted context omitted.

Yep. That said, unlike cheetahs, there’s plenty of evidence of leopards attacking humans. And these days, it’s the leopards, the closed-AI types and misanthropes -- telling everyone, “AI will take your job and there’s nothing you can do about it.”

I keep seeing this and I want to speak to it. When Dario and others say things like "this is happening and we should probably figure out what to do about it" what ends up happening is people hear "this is happening," see that the person warning them is the person doing the thing, and then short-circuit. "Why can't you just stop then?" Dario's point, and the point of the people actually trying to solve the problem, is…

Physics allows this, and actually taking advantage of it requires billions of dollars of unprecedented infrastructure buildout that is already destabilizing the power grid.

The only reason that infrastructure buildout is happening at all is the ideological capture of a small handful of obscenely wealthy people, who are fueling this buildout by spreading the extreme paranoia you’re echoing here.

I do not understand why no one else can see the circularity of this reasoning. There is nothing inevitable about tying up all of this productive capital in the pursuit of AI. There are many, many other projects requiring similar capital and human effort, with much more obvious payoffs, such as decarbonizing the world’s energy systems.

“It’s physically possible to provide abundant electricity without burning fossil fuels” is more provably true than any of the insane science fiction bullshit that undergirds the AI buildout, and yet, the entire clean energy industry is still having to build insane financial Rube Goldberg contraptions to make incremental progress.

“Inevitability” is a lie, period. This entire thing is extremely historically contingent, and we could easily stop this train tomorrow.

Re: Superintelligence: The Idea That Eats Smart People (2016)

#64

Post by smaudet »

stared wrote: post #53

I like this essay, and have very often referred to it when someone talks about AGI. There is a common narrative bias to look at AGI as the Abrahamic God, if not explicitly, then just by saying that it is omniscient, omnipotent, immortal - and will judge us for our deeds. It is tempting for anyone raised in the West, and immersed in Judeo-Christian culture. And for anyone, in general, as it offers an epic narration of…

For all its talk of inoculation, this is a terribly written essay. They do not make a point, nor even arguments, instead, opting to ramble in hopes that you forget whatever it was you were thinking.

The issue is simple. Just like us (who are arguably complex, look at what we're building over here, this AI computer stuff!), entities have simple core needs (like food, water, power, etc.).

An infinitely smart AGI has the potential, nay, likely cause, to require infinite resources. We're already seeing the effect in the computing sector on e.g. chips, there's no reason to think this trend won't continue...

Lets circle back to the hydrogen argument, will we blow ourselves up. Real concern, abated by hard numbers. Different atmosphere, different concentrations, different pressure, different possible outcomes.

Today, we don't have those numbers. We don't have those calculations. I don't disagree with the point at the end "about how people can exploit other people, or through carelessness introduce immoral behavior into automated systems". These are issues, too. But saying there are other issues, don't worry about this big issue over here, is the absolute worse argument possible.

That's hand waving.

Re: Superintelligence: The Idea That Eats Smart People (2016)

#65

Post by notahacker »

aqfamnzc wrote: post #49
shagie wrote: post #27

Earlier quoted context omitted.

> Cheetahs are very fast, but humans have way more endurance. https://en.wikipedia.org/wiki/Man_versus_Horse_Marathon > The Man versus Horse Marathon is an annual race over 21 miles (34 km), where runners compete against riders on horseback through a mix of road, trail and mountainous terrain. The race, which is a shorter distance than an official marathon road race, takes place in the Welsh town of Llanwrtyd Wells e…

Wow, seems surprisingly balanced. I would guess that if it was 50% longer distance than humans would win reliably, and 50% shorter would allow the horses to win?

Think it's more to do with hilly, wooded courses unfavourable to horses and a requirement that the horse's heart rate remain below a certain threshold when inspected by a vet in the middle and end of a course, whereas the leading humans are maxing out their efforts

I don't think the Mongol cavalry would lose races to humans over any distance of steppe

Re: Superintelligence: The Idea That Eats Smart People (2016)

#67

Post by stoneman24 »

foobiekr wrote: post #38
topkai22 wrote: post #26

Earlier quoted context omitted.

I think the term is persistence hunting ( https://en.wikipedia.org/wiki/Persistence_hunting ). There was an intriguing blog post years ago (which sadly I can’t find) about how terrifying a fictional portrayal of persistence hunting would be- grinding down prey through exhaustion, thinking you’ve escaped but being slowly tracked down.

There's also the story "Go, Go, Go, Said the Bird" by Sonya Dorman in the first Dangerous Visions which is like this, in a way.

There’s also “the ruum” by Arthur Porges[0]. We got as part of English class in high school, a long time ago.

Try not to read the Wikipedia as it might spoil the short story, there’s the pdf available on the web somewhere

[0] https://en.wikipedia.org/wiki/The_Ruum

Re: Superintelligence: The Idea That Eats Smart People (2016)

#68

Post by hilariously »

braden-lk wrote: post #41
andai wrote: post #14

Has anyone played SOMA? Spoiler warning. It explores this idea of, what if there's an AI in charge of ensuring mankind survives at all costs. What would it be willing to do, to keep us alive? Would we even recognize the result as human? It's a horror game and it explores all kinds of fascinating and disturbing scenarios. Simulations of human minds. Artificial worlds. Human minds in robot bodies. Genetically modified…

One of my favorite games, and I recommend it to anyone who loves a good existential sci-fi horror. If you are not comfortable with stealth games, it also has a "Safe Mode" where enemies aren't a threat to you if you just want to experience the story. Spoiler warning for those that havent played-- I forget the details exactly, but one scene stuck with me. It was a screen in one of the labs, where an experiment was run…

There's definitely a short story in Accelerando about this same topic.

Re: Superintelligence: The Idea That Eats Smart People (2016)

#70

Post by ACCount37 »

blamestross wrote: post #6

AI Superintelligence doesn't scare me for the same reasons "grey goo" doesn't scare me. We are awash in self-replicating machines. The biosphere is already a grey-goo apocalypse. Any new competitors have a serious moat to cross to out compete any existing self-replicators. We are awash in intelligent agents. Our society (and meta society) is full of superhuman agents already. There is a huge moat for any new intellig…

"Intelligent agents" we have around run on a metabolic budget of 25W and a hardware platform the size of a melon.

Human intelligence doesn't scale upwards well. Individual humans only get this smart, and there are gains from getting multiple humans to work together - but the more of them you add, the larger is your communication and coordination overhead. In no small part because humans are self-interested agents that simply aren't designed to compose their capabilities seamlessly. You can't get a vastly superhuman intelligence simply by piling together more humans.

Human intelligence doesn't scale sideways well either. Unskilled labor is cheap and plentiful, but if you have a human with a very specific skill, the process of getting more of that capability is very long and very involved. Often, it's easier to redesign an entire process to run on worse humans than it is to train more humans for better performance.

Institutions are more capable than individuals, but far less capable than the sum of individuals within them. At many corporations, the majority of individual productivity is absorbed by management overhead and corporate rot.

AI isn't bounded by those limitations.

AI can scale intensively and extensively. AI can be scaled up by upping the compute budgets. AI can be replicated and copied indefinitely. AI doesn't have the innate human "I don't live to work, I work to live" overhead. AI can outclass human intelligence by a long shot.

The "moat" that's there is already being eroded by modern day LLMs. Betting that future AI systems can't cross it is folly.

Post reply on HN