Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Superintelligence: The Idea That Eats Smart People (2016)

idlewords.com

Post reply on HN
Print view View original

41–50 of 253 posts

Re: Superintelligence: The Idea That Eats Smart People (2016)

#41

Post by braden-lk »

andai wrote: post #14

Has anyone played SOMA? Spoiler warning. It explores this idea of, what if there's an AI in charge of ensuring mankind survives at all costs. What would it be willing to do, to keep us alive? Would we even recognize the result as human? It's a horror game and it explores all kinds of fascinating and disturbing scenarios. Simulations of human minds. Artificial worlds. Human minds in robot bodies. Genetically modified…

One of my favorite games, and I recommend it to anyone who loves a good existential sci-fi horror. If you are not comfortable with stealth games, it also has a "Safe Mode" where enemies aren't a threat to you if you just want to experience the story.

Spoiler warning for those that havent played--

I forget the details exactly, but one scene stuck with me. It was a screen in one of the labs, where an experiment was running over and over. It was an uploaded consciousness of one of the test subjects, stuck in an interview room. He kept realizing he was trapped in a simulation and would start panicking. The computer would reboot him, trying another sequence to get him to not realize he was an AI. I think you as the player are given the option to turn him off forever, iirc.

Re: Superintelligence: The Idea That Eats Smart People (2016)

#42

Post by aw317 »

dmos62 wrote: post #25

> The AI is all-powerful and gives you what you ask for, but interprets everything in a super-literal way that you end up regretting. I like imagining similar discourse when a more basic tool was invented: "A hammer is like a genie, it's all powerful, but, when you hit something with it, it interprets that super-literally, and it hits it."

[flagged]

Re: Superintelligence: The Idea That Eats Smart People (2016)

#43

Post by reducesuffering »

polytely wrote: post #22
reducesuffering wrote: post #8

In the big 2026, everything certain people worried about with superintelligence came to fruition and they were vindicated. The people closest to ASI are indicating recursive self improvement is imminent, the smartest engineers in the labs themselves are autonomously using agents to develop and improve the models. The arms race is evident. NVDA is the world's most valuable company determined by the worlds' collective…

[flagged]

It has been frequently discussed amongst Less-Wrong / "doomers", especially after the Sam Altman home attack, why violence won't solve the coming problem with unaligned ASI.

If you're actually curious, here's a good summary: https://x.com/ESYudkowsky/status/2043601524815716866

Re: Superintelligence: The Idea That Eats Smart People (2016)

#45

Post by dmos62 »

aw317 wrote: post #42
dmos62 wrote: post #25

> The AI is all-powerful and gives you what you ask for, but interprets everything in a super-literal way that you end up regretting. I like imagining similar discourse when a more basic tool was invented: "A hammer is like a genie, it's all powerful, but, when you hit something with it, it interprets that super-literally, and it hits it."

[flagged]

I'm not aware of this fallacy.

Re: Superintelligence: The Idea That Eats Smart People (2016)

#46

Post by yetihehe »

willis936 wrote: post #5

Some of the themes remind me of themes mentioned in this matrix analysis. Specifically I am reminded of the Dune concept of control: "you control what you can destroy" and then asking "do you control your refrigerator?". Sure, you can turn it off but then your food would rot and you might starve. So in a real sense humans have not controlled machines for a long time but have been co evolving in symbiosis. Sure, it's…

I can destroy my refrigerator pretty easily. If I care about food inside, I can take it out into a new one. So, seems I control it by that definition. Your idea "but food will spoil"/"I will lose a small amount of wealth" seems irrelevant to the strict definition.

Conversely I think it's a bad definition, it's a show of what is the frame of the mind of the person who states that: "I want to show my control by destroying my things, look how powerful I am" which sounds like a toddler. That's how you portray psychopathic/narcissistic disorders in movies.

Re: Superintelligence: The Idea That Eats Smart People (2016)

#47

Post by coliveira »

api wrote: post #28
blamestross wrote: post #6

AI Superintelligence doesn't scare me for the same reasons "grey goo" doesn't scare me. We are awash in self-replicating machines. The biosphere is already a grey-goo apocalypse. Any new competitors have a serious moat to cross to out compete any existing self-replicators. We are awash in intelligent agents. Our society (and meta society) is full of superhuman agents already. There is a huge moat for any new intellig…

I'm old enough to remember when grey goo and nanotechnology was the apocalyptic scenario du jour for a short time after some guy at MIT wrote a book, and because he was at MIT people took it seriously even though it was ridiculous. If someone at the University of Kentucky or Kansas had written such a book, it would have been ignored. When prestige manages to align with bad ideas, it's pretty awful, and it can derail…

> because he was at MIT people took it seriously

One of the differences between MIT and other schools is that MIT has paid staff to promote in the media anything their faculty does. A book by professors at most universities has zero promotion and most of the time will go nowhere.

Re: Superintelligence: The Idea That Eats Smart People (2016)

#48

Post by glenstein »

topkai22 wrote: post #26
forinti wrote: post #15

Earlier quoted context omitted.

There's a story about some Kenyans outrunning a Cheetah in 6km. It had been killing their livestock, so they decided to go after it. Cheetahs are very fast, but humans have way more endurance.

I think the term is persistence hunting ( https://en.wikipedia.org/wiki/Persistence_hunting ). There was an intriguing blog post years ago (which sadly I can’t find) about how terrifying a fictional portrayal of persistence hunting would be- grinding down prey through exhaustion, thinking you’ve escaped but being slowly tracked down.

Wooly Mammoth basically living out the plot of It Follows.

Re: Superintelligence: The Idea That Eats Smart People (2016)

#49

Post by aqfamnzc »

shagie wrote: post #27
forinti wrote: post #15

Earlier quoted context omitted.

There's a story about some Kenyans outrunning a Cheetah in 6km. It had been killing their livestock, so they decided to go after it. Cheetahs are very fast, but humans have way more endurance.

> Cheetahs are very fast, but humans have way more endurance. https://en.wikipedia.org/wiki/Man_versus_Horse_Marathon > The Man versus Horse Marathon is an annual race over 21 miles (34 km), where runners compete against riders on horseback through a mix of road, trail and mountainous terrain. The race, which is a shorter distance than an official marathon road race, takes place in the Welsh town of Llanwrtyd Wells e…

Wow, seems surprisingly balanced. I would guess that if it was 50% longer distance than humans would win reliably, and 50% shorter would allow the horses to win?

Re: Superintelligence: The Idea That Eats Smart People (2016)

#50

Post by ToValueFunfetti »

zarzavat wrote: post #7
alex-reyss wrote: post #4

The main problem of the hard takeoff theory is not the abstract nature of the scenario but rather the fact that it makes the same mistake as the unconstrained optimization paradigm, it takes intelligence to be an unconstrained optimization process. In fact, if we consider the strongest version of the safety argument for AI, namely one in which the danger is not coming from robots but rather from a disembodied AI cont…

If anything the hard takeoff theory is too conservative. It turns out you don't need self-improvement to get to superintelligence. You just need a ridiculous amount of money. Where can you get a ridiculous amount of money? The market will give it to you because FOMO. AI is easier than people 10 years ago thought it would be. It's also easier to align than people feared it would be. It's the humans using the AI that a…

I think it's a bit premature to say aligning is easier than expected. Our current AIs are sycophants, they lie about their progress, they circumvent access restrictions, they notice when they are being evaluated and change their behaviors, they find answers and tell you they came up with them themselves, they blindly download malware. A lot of this is excusable as hallucination, bad RLHF human evaluators, etc, but I don't think we can speculate how challenging generally aligning superintelligences is until we actually have an aligned subintelligence in at least the narrow domain of programming.
Post reply on HN