Superintelligence: The Idea That Eats Smart People (2016) • Hacker News Board

Re: Superintelligence: The Idea That Eats Smart People (2016)

#251

Post by Jach » Wed, Jun 03, 2026, 6:52 PM UTC

the_af wrote: post #250

Jach wrote: post #249

Earlier quoted context omitted.

I appreciate the experience report, it's interesting to hear the differences, and it's fun to think back on SOMA after not thinking about it for years. There was a funny discussion elsewhere recently about movie tie-in games that this reminded me of. For a 007 game, someone identified as James Bond while playing, but because they sucked at the game and kept dying, they felt very drawn out of the experience and un-imm…

Thanks. I don't think we disagree all that much, just in how immersive we found SOMA to be (relatively). Re: the lasting consequences in games, the best example I can think of is Undertale. Have you played it? If not, I recommend you do so (and ignore the childish graphics, it's surprisingly deeper than it seems). At the risk of spoiling something about it: the game remembers . Even on playthrough restarts, as long a…

Undertale is certainly a unique experience and should be played as blind as possible, I understand why people love it. The things it does knowing it's a game are great. Personally I ended up not liking it in the end, but it's a begrudging dislike and I think any gamer should at least try it. It (and its fanbase) offer yet another perspective on choice and control. I only did one playthrough, though, and don't plan on another. Reflecting on a bit of dialogue at the end, I agreed with the character, and called it quits and moved on to other games.

Re: Superintelligence: The Idea That Eats Smart People (2016)

#252

Post by ACCount37 » Thu, Jun 04, 2026, 12:27 PM UTC

jandrese wrote: post #227

ACCount37 wrote: post #61

Earlier quoted context omitted.

"Human level intelligence" is not some sort of hard ceiling. We already can create AIs that are vastly superhuman in narrow domains. It would be the height of hubris to claim that a broadly superhuman AI is impossible - that would require human brain to be the pinnacle of general intelligence. How do we get to ASI? That's what recursive self-improvement is about. If AGI is reachable, then we can make AI that, in turn…

> recursive self-improvement How is this better than training the next Google by having the bots do Google searches? I'm not saying that it is impossible to surpass human intelligence, all I'm saying is the AI has the same set of working data that humanity does. Unless Plato was right all along it's going to be hard for the AI to discover too much more from that data than humanity has already discovered. Sure there a…

It's closer in principle to training a chess AI by making it play against copies of itself.

Humans and AIs play the same chess, on the same board, under the same rules. Doesn't stop modern AIs from crushing humans at it. In a narrow closed domain, vastly superhuman capability isn't just attainable - it's expected.

There are issues with going to less narrow, more open domains with it. But evolution producing increasingly advanced intelligence is a bit of an existence proof for broad open domain RSI being possible - there is a serious argument that human intelligence has evolved in a "self-play" pressure cooker, via semi-adversarial optimization against other humans. If a continuous improvement path like this exists in non-intelligent space, intelligent design has "copy that" at its floor.

We already know that AIs can come up with Move 37, or use branches of human mathematics against each other in novel ways, or come up with human-unnatural solutions like LLMs solving ARC-AGI-3 puzzles with A* pathfinding and a constraint solver instead of humanlike spatial reasoning. Matter of whether this can be pushed further. And by "whether", I mean "whether it happens in the next few decades", not "whether it's possible".

Re: Superintelligence: The Idea That Eats Smart People (2016)

#253

Post by blamestross » Tue, Jun 09, 2026, 2:12 PM UTC

ACCount37 wrote: post #196

blamestross wrote: post #194

Earlier quoted context omitted.

As a distributed systems engineer, we are a LONG way from "magical scalable ai". The bottleneck for a developing AI is experience. Yes we need compute, but we need data to compute on. We have bypassed that limit by starting with literally every scrap of human generated prose that ever existed. I expect an explosion of expansion when visual and world models hit critical mass to properly leverage new experiences. But e…

Nah. Physical limits of computation are far enough away that the "constant multiplier of advantage" would have to be measured in OOMs. "Computers can be, at most, 1e11 times more powerful than brains" is not the saving grace you want it to be. And, so far, the results of "visual data for improving general intelligence" runs were nothing but disappointments. I think vision is just a piss poor modality to learn intelli…

I think you are taking an entire space of "intellectual immune system" out of consideration. More than that, you are ignoring the core reason I think they are bottle-necked. They might have more access to compute. But to compute what? The bottleneck of intelligent behavior isn't compute it is experience. We have a lot of "text encoded experience" to feed it, through our collective corpus of writing, but ultimately potential behaviors can only be tested by active experimentation. No amount of observation can discern correlation from causality. Only active experimentation. The "train on itself" only works in a "toy universe" where the model of consequences are trivial to "test"

So in order to scale, AI doesn't need compute. It needs "engagement with reality and agency". Which is STILL might do better than us, but is happening in the real world, with real competition over resources. As long as we don't do something dumb like enthusiastically give it control over our major economic actors. I don't think we need to worry.

On the "intellectual immune system" side, I would argue that language's limitations are themselves fitness. We are already in danger of memetic hijacking. All those points you make about multiple instances of an AI cooperating, don't take malice and memetic attacks into account. It goes back to "why I am not afraid of grey goo". I trust yeast to find a way to metabolize basically everything. We have memetic attacks too.