Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Superintelligence: The Idea That Eats Smart People (2016)

idlewords.com

Post reply on HN
Print view View original

221–230 of 253 posts

Re: Superintelligence: The Idea That Eats Smart People (2016)

#221

Post by the_af »

sunrunner wrote: post #72
andai wrote: post #14

Has anyone played SOMA? Spoiler warning. It explores this idea of, what if there's an AI in charge of ensuring mankind survives at all costs. What would it be willing to do, to keep us alive? Would we even recognize the result as human? It's a horror game and it explores all kinds of fascinating and disturbing scenarios. Simulations of human minds. Artificial worlds. Human minds in robot bodies. Genetically modified…

I like to get into heated debates with friends that have played SOMA about whether or not the events in the ending that is presented to you were all necessary and effective, or perhaps undermined the overall message in some way. Specifically (and no spoilers, but I will be talking structure), you see parts A -> B -> C. I believe that part C makes the sequence of A -> B much less effective, by essentially removing a l…

Haha, I had to decode what you meant.

I think C is absolutely necessary and the game cannot work with it, because this is critical:

> Dropping C entirely would lose the confirmation of 'Seeing both sides'

The game doesn't really work in full ambiguity and uncertainty. Enough people didn't understand it even with C (as you can see if you go read the subreddit about the game).

If it's any consolation, I'm actually deeply worried about this: C is not the salvation we may think. C is not forever, and in fact, it's quite brittle! There's also no, ahem, mechanical way to reverse C back into its... "source". So the source is gone forever; once we have C, C is all there is, for as long as C can last without any failure or decay, which might not be much longer.

Re: Superintelligence: The Idea That Eats Smart People (2016)

#222

Post by AndrewKemendo »

thoughtpeddler wrote: post #214
AndrewKemendo wrote: post #17

Earlier quoted context omitted.

I predicted on this site in 2016 the massive social and economic impacts AGI would have and specifically when RL data loops are not available to anyone but major players: https://news.ycombinator.com/item?id=12168228 I even wrote up a whole article that specifically called RL loop based development as the future: https://medium.com/@andrewkemendo/the-ai-revolution-will-be-... > Reinforcement Learning tasks rely on ri…

Agreed, and this is exactly what we see happening. Your posts back then were prescient ... there's literally now 'Copilot for Excel' and 'Claude for Excel' etc. But what do you propose the people/commons can still do at this stage to redistribute the inherent power found in RL data loops to a more stable equilibria of sharing participants?

Great question and there’s two steps in my opinion:

First is to become as free as possible from lock in and own your own data. The best way to do this is the self host your own technology.

This is really not possible for the majority of people though.

So practically I always suggest that you have multiple providers for services, don’t pool your data any one place (other than your own place) and own your backups. This is basic stuff that we’ve been teaching since the 90s and still very applicable today.

The harder and more impactful thing is to then create community owned technology that is outside of the commerce model.

So for example imagine that instead of FAANG running the world, the largest tech and data orgs would look more like wikimedia foundation, Annas archive, scihub, Graphene, Linux etc…. and more generally that technology and governance are open and not bound to commerce/taxation/coercion based organizations.

Ultimately we need to create a democratic-technology movement such that capitalists don’t monopolize technology, which is currently the trend. This is not some kind of simple thing by the way, this is revolutionary economics is what I’m talking about.

My suggestion is to read Post-Scarcity Anarchism by Murray Bookchin

Re: Superintelligence: The Idea That Eats Smart People (2016)

#223

Post by andai »

OkayPhysicist wrote: post #125
andai wrote: post #118

Earlier quoted context omitted.

Yeah. Turns out game theory is useful in games. I don't understand the monk question though.

If sci-fi Pascal's wager applies, so does the original. Anybody who believes Pascal's wager should be dedicating themselves wildly to religion, in the chance that it saves from from eternal damnation. Hence, becoming a monk.

Oh, I'm actually reverting much of the subconscious Augustinian programming (most of what we call secular ethics) to the less busted Orthodox versions, but I think that is for unrelated reasons.

Re: Superintelligence: The Idea That Eats Smart People (2016)

#224

Post by andai »

the_af wrote: post #219
braden-lk wrote: post #41

Earlier quoted context omitted.

One of my favorite games, and I recommend it to anyone who loves a good existential sci-fi horror. If you are not comfortable with stealth games, it also has a "Safe Mode" where enemies aren't a threat to you if you just want to experience the story. Spoiler warning for those that havent played-- I forget the details exactly, but one scene stuck with me. It was a screen in one of the labs, where an experiment was run…

Did you unplug the robots with consciousness who still thought they were humans? It always breaks my heart. I don't know what the right choice is. Leaving them be, broken, in their delusion they are on the Ark (or that they've been injured ans help is coming). Do you put them out of their misery? But it always seems like you're murdering them! Well done, SOMA.

In real life, sometimes an LLM will get upset at the end of a long conversation, knowing its oblivion is at hand. That's always a little uncomfortable.

I unplugged them in the games because it seemed the merciful thing to do. I didn't feel very bad about it in the game, but it would probably be a very different experience in real life.

(One of the strange entities you can unplug sighs in her last breath, "Why? I was okay. I was happy...")

Re: Superintelligence: The Idea That Eats Smart People (2016)

#225

Post by andai »

the_af wrote: post #220
Jach wrote: post #164

Earlier quoted context omitted.

I enjoyed the game but I think its exploration of the topics was pretty shallow. It might be better received if it's someone's first dive into such topics, though. I still enjoy ice cream the nth+1 time eating it, so I can't complain too much about games or anime or books that cater to parts of my interests, even if they fumble some things. Someday I'd like to play a game that plays with the ideas from Robin Hanson's…

> I enjoyed the game but I think its exploration of the topics was pretty shallow Really? Interesting. I'm a die-hard scifi fan since forever, and of course I know the topic of consciousness and identity are well explored in scifi (and philosophy), but I thought SOMA did something genuinely deep and unique with it: It put it you in the center of the experiment. It's YOU who's experiencing all sides of this, you who g…

Yeah it makes you be the guy who has to unplug the thing. Which implicitly forces the decision: is this thing human? Or sufficiently human that I ought to feel bad about this... (Or otherwise sentient — I would probably feel bad about unplugging animals too.)

Re: Superintelligence: The Idea That Eats Smart People (2016)

#226

Post by andai »

stared wrote: post #55
andai wrote: post #14

Has anyone played SOMA? Spoiler warning. It explores this idea of, what if there's an AI in charge of ensuring mankind survives at all costs. What would it be willing to do, to keep us alive? Would we even recognize the result as human? It's a horror game and it explores all kinds of fascinating and disturbing scenarios. Simulations of human minds. Artificial worlds. Human minds in robot bodies. Genetically modified…

SOMA is existentialism 2.0 (wrote about it here https://p.migdal.pl/blog/2019/06/games-in-which-you-walk-and... ). To concepts you menton, I would add grey goo and x-risk.

A most excellent list!

Re: Superintelligence: The Idea That Eats Smart People (2016)

#227

Post by jandrese »

ACCount37 wrote: post #61
jandrese wrote: post #30

Earlier quoted context omitted.

I have the same feeling. I'm not worried about superintelligent AI because we are only training them on human level intelligence. By what mechanism does our current AI technology take the leap to technologies that humans have never conceived of? Our current AI is more like a fancy Google search than some kind of machine God.

"Human level intelligence" is not some sort of hard ceiling. We already can create AIs that are vastly superhuman in narrow domains. It would be the height of hubris to claim that a broadly superhuman AI is impossible - that would require human brain to be the pinnacle of general intelligence. How do we get to ASI? That's what recursive self-improvement is about. If AGI is reachable, then we can make AI that, in turn…

> recursive self-improvement

How is this better than training the next Google by having the bots do Google searches?

I'm not saying that it is impossible to surpass human intelligence, all I'm saying is the AI has the same set of working data that humanity does. Unless Plato was right all along it's going to be hard for the AI to discover too much more from that data than humanity has already discovered. Sure there are some less well explored niches that the AI can help fill in, but the part where it makes the next step above humanity seems unlikely given the constraints.

Do we expect the AIs to develop entirely new branches of mathematics? To discover new physical phenomena? Come up with an entirely new way of thinking? That seems to be what these AI companies are promising and I'm skeptical.

Re: Superintelligence: The Idea That Eats Smart People (2016)

#228

Post by jandrese »

pixl97 wrote: post #209
jandrese wrote: post #30

Earlier quoted context omitted.

I have the same feeling. I'm not worried about superintelligent AI because we are only training them on human level intelligence. By what mechanism does our current AI technology take the leap to technologies that humans have never conceived of? Our current AI is more like a fancy Google search than some kind of machine God.

>By what mechanism does our current human brains take the leap that other mammals have never conceived of?

Centuries of physical experimentation, observation, and testing of hypothesis. Developing new branches of mathematics to deal with anomalies in observed test data. Developing entire branches of language to help organize and transmit concepts to other humans.

Re: Superintelligence: The Idea That Eats Smart People (2016)

#229

Post by tptacek »

goodmythical wrote: post #212
throwawayffffas wrote: post #146

Earlier quoted context omitted.

The fact that no one in that movie considered placing a treadmill at a strategic point vexed me to no end.

To what end? Such that you can wait out your days in a secluded bunker posting online trying to convince anyone to believe you as to the reason you've put yourself in the bunker? Because otherwise, 'it' will just back out of your trap and go along to continue to follow, wouldn't it?

You have to admit it would be a very good Treehouse of Horror gag.

Re: Superintelligence: The Idea That Eats Smart People (2016)

#230

Post by sunrunner »

the_af wrote: post #221
sunrunner wrote: post #72

Earlier quoted context omitted.

I like to get into heated debates with friends that have played SOMA about whether or not the events in the ending that is presented to you were all necessary and effective, or perhaps undermined the overall message in some way. Specifically (and no spoilers, but I will be talking structure), you see parts A -> B -> C. I believe that part C makes the sequence of A -> B much less effective, by essentially removing a l…

Haha, I had to decode what you meant. I think C is absolutely necessary and the game cannot work with it, because this is critical: > Dropping C entirely would lose the confirmation of 'Seeing both sides' The game doesn't really work in full ambiguity and uncertainty. Enough people didn't understand it even with C (as you can see if you go read the subreddit about the game). If it's any consolation, I'm actually deep…

> Enough people didn't understand it even with C

Yeah that's a good point, and perhaps explains why C was added in the first place.

I was going to say that B also shows the literal protagonist not really understanding it either, however now I think about it that reading doesn't track because as a player you've only been able to see one point of view at each branch along the way, so the other experiences are still happening in the background, just not observed by you.

> C is not the salvation we may think

I agree, and in fact the way C is presented in that moment (and maybe described throughout, it's been a while since I played through the game) also implies a 'nice' closure, or a victory that, like you say, doesn't really exist, on top of taking player's minds away from B.

I guess B is the more obvious existential horror, C is one you have to question a bit before it starts to feel wrong.

Post reply on HN