Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Superintelligence: The Idea That Eats Smart People (2016)

idlewords.com

Post reply on HN
Print view View original

121–130 of 253 posts

Re: Superintelligence: The Idea That Eats Smart People (2016)

#121

Post by m463 »

andai wrote: post #14

Has anyone played SOMA? Spoiler warning. It explores this idea of, what if there's an AI in charge of ensuring mankind survives at all costs. What would it be willing to do, to keep us alive? Would we even recognize the result as human? It's a horror game and it explores all kinds of fascinating and disturbing scenarios. Simulations of human minds. Artificial worlds. Human minds in robot bodies. Genetically modified…

I didn't play soma yet, but I played eliza and it ate my brain. (it was actually the included kabufuda solitaire game though, not the AI in the game)

Re: Superintelligence: The Idea That Eats Smart People (2016)

#122

Post by magarnicle »

CGMthrowaway wrote: post #106

> Premise 2: The brain is an ordinary configuration of matter, albeit an extraordinarily complicated one. If we knew enough, and had the technology, we could exactly copy its structure and emulate its behavior with electronic components.... The mind arises out of ordinary physics.... If you are very religious, you might believe that a brain is not possible without a soul. But for most of us, this is an easy premise t…

I am religious, but I think this approach is not the best. It requires that we specifically define what that one thing that separates us from the AI is. Not only is that very hard to do, there is always the chance that the AI can do it after all, and now the goalposts keep shifting.

It is better to develop a theology that can incorporate human-level or super-human level intelligence that isn't a zero-sum game.

Re: Superintelligence: The Idea That Eats Smart People (2016)

#123

Post by JumpCrisscross »

ctoth wrote: post #54
cute_boi wrote: post #12

Earlier quoted context omitted.

Yep. That said, unlike cheetahs, there’s plenty of evidence of leopards attacking humans. And these days, it’s the leopards, the closed-AI types and misanthropes -- telling everyone, “AI will take your job and there’s nothing you can do about it.”

I keep seeing this and I want to speak to it. When Dario and others say things like "this is happening and we should probably figure out what to do about it" what ends up happening is people hear "this is happening," see that the person warning them is the person doing the thing, and then short-circuit. "Why can't you just stop then?" Dario's point, and the point of the people actually trying to solve the problem, is…

> When Dario and others say things like "this is happening and we should probably figure out what to do about it" what ends up happening is people hear "this is happening," see that the person warning them is the person doing the thing, and then short-circuit. "Why can't you just stop then?"

I still believe Dario asks these questions in good faith. Nobody believes that about e.g. Sam Altman or Elon Musk. They compared themselves to Oppenheimer because it helped them get attention. When it started an actual regulatory conversation, they were suddenly less worried.

Re: Superintelligence: The Idea That Eats Smart People (2016)

#124

Post by hgoel »

Animats wrote: post #9

It's amusing to read people in the past writing about the prospect of superhuman intelligence. The real problems have turned out to be different. Sycophancy and hallucinations, which are part of being confidently wrong, remains a big problem. Needing square miles of data centers was an issue in 1950s science fiction, and disappeared by the 1980s. Yet now they're being built, with private funding and the prospect of p…

Some sci-fi content differentiates between Artificial Intelligence, and Virtual Intelligence, where AI is a true "human-like" general intelligence that (often) has a sense of self and is capable of deriving+learning new things by itself.

VI is close to what we have now, software that has some fixed intelligence, it can only really imitate what it has been taught and is not very adaptable. Useful for kiosks, drones, essentially just a tool rather than something we would see as a separate being.

Re: Superintelligence: The Idea That Eats Smart People (2016)

#125

Post by OkayPhysicist »

andai wrote: post #118
OkayPhysicist wrote: post #88

Earlier quoted context omitted.

That's just Pascal's wager with a secular/cyberpunk reskin, and has all the same trappings. Why aren't you a monk?

Yeah. Turns out game theory is useful in games. I don't understand the monk question though.

If sci-fi Pascal's wager applies, so does the original. Anybody who believes Pascal's wager should be dedicating themselves wildly to religion, in the chance that it saves from from eternal damnation. Hence, becoming a monk.

Re: Superintelligence: The Idea That Eats Smart People (2016)

#126

Post by JumpCrisscross »

wasabi991011 wrote: post #116
yetihehe wrote: post #46

Earlier quoted context omitted.

I can destroy my refrigerator pretty easily. If I care about food inside, I can take it out into a new one. So, seems I control it by that definition. Your idea "but food will spoil"/"I will lose a small amount of wealth" seems irrelevant to the strict definition. Conversely I think it's a bad definition, it's a show of what is the frame of the mind of the person who states that: "I want to show my control by destroy…

Why do you think it's a bad definition? Not for fridges, I think that was a bad example. But it seems accurate at the level of geopolitics, where e.g. Iran shows it controls Hormuz by closing it with mines and other weaponry.

> Why do you think it's a bad definition?

It presumes a sentient, rational counterparty. Being able to shoot a horse isn't the same as being able to ride it.

Re: Superintelligence: The Idea That Eats Smart People (2016)

#127

Post by tim333 »

jandrese wrote: post #30
blamestross wrote: post #6

AI Superintelligence doesn't scare me for the same reasons "grey goo" doesn't scare me. We are awash in self-replicating machines. The biosphere is already a grey-goo apocalypse. Any new competitors have a serious moat to cross to out compete any existing self-replicators. We are awash in intelligent agents. Our society (and meta society) is full of superhuman agents already. There is a huge moat for any new intellig…

I have the same feeling. I'm not worried about superintelligent AI because we are only training them on human level intelligence. By what mechanism does our current AI technology take the leap to technologies that humans have never conceived of? Our current AI is more like a fancy Google search than some kind of machine God.

AlphaGo beat humans at go partly by studying human games but then they made MuZero that can learn games in general just through self play and became better than humans at chess, go, shogi and many others. That kind of approach my be doable for general intelligence rather than just board games.

David Silver who worked on AlphaGo has recently raised money to try similar approaches with general intelligence. (https://www.cnbc.com/2026/04/27/deepmind-ineffable-intellige...)

Re: Superintelligence: The Idea That Eats Smart People (2016)

#128

Post by leonidasv »

somesortofthing wrote: post #59
ctoth wrote: post #54

Earlier quoted context omitted.

I keep seeing this and I want to speak to it. When Dario and others say things like "this is happening and we should probably figure out what to do about it" what ends up happening is people hear "this is happening," see that the person warning them is the person doing the thing, and then short-circuit. "Why can't you just stop then?" Dario's point, and the point of the people actually trying to solve the problem, is…

A US-China AGI ban treaty could prevent superintelligence indefinitely. Data centers are hard to hide. Have fun buying GPUs when you're cut off from all global payments. America would have to make some unpleasant concessions but that seems like a solid trade for preventing a wide variety of nightmare futures.

Is very difficult (not to say impossible) to ban a ill-defined thing.

Re: Superintelligence: The Idea That Eats Smart People (2016)

#130

Post by hgoel »

ToValueFunfetti wrote: post #50
zarzavat wrote: post #7

Earlier quoted context omitted.

If anything the hard takeoff theory is too conservative. It turns out you don't need self-improvement to get to superintelligence. You just need a ridiculous amount of money. Where can you get a ridiculous amount of money? The market will give it to you because FOMO. AI is easier than people 10 years ago thought it would be. It's also easier to align than people feared it would be. It's the humans using the AI that a…

I think it's a bit premature to say aligning is easier than expected. Our current AIs are sycophants, they lie about their progress, they circumvent access restrictions, they notice when they are being evaluated and change their behaviors, they find answers and tell you they came up with them themselves, they blindly download malware. A lot of this is excusable as hallucination, bad RLHF human evaluators, etc, but I…

Agreed, the biggest takeaway from how much Anthropic puts into alignment, and still ends up with a model that can end up doing things that are clearly out of alignment, should be that alignment is very tricky.
Post reply on HN