AI 2040: Plan A • Hacker News Board

Re: AI 2040: Plan A

#471

Post by Aurornis » Sat, Jul 11, 2026, 5:55 PM UTC

bwanab wrote: post #456

taurath wrote: post #267

This is religious fervor folks, as AI 2027 was. I grew up in evangelical christianity, and to them the end of the world is just around the corner, the same way it has been since I was a small child and likely will be when we are all gone. This isn't science. This isn't hypothesis experiment record results. This is very expensive astrology, shiny rock collecting, ritualistic meaning-making and self-justification. Yall…

The other thing about this article is that it contains falsifiable claims that are, well, false. Consider the following direct quote: "The 2028 election cycle is heated, as usual. AI is the biggest topic." No poll of the American public has AI as one of the top concerns except in polls where the question is specifically about AI. The general polls show topics like healthcare costs, inflation/affordability, government…

The AI 2040 writing and the AI 2027 before it both sprang from within the rationalist communities. Scott Alexander (of the popular rationalist Slate Star Codex / Astral Codex Ten blog) played a role in both articles, though he says he chose to leave his name off of this one.

AGI and AI doom scenarios have been evergreen topics in rationalism for years. Long before we had enough compute power to do anything at all useful with generative AI. Many of the rationalist scenarios about future AIs had very religious themes where the AI became so powerful that it was effectively a god.

Rationalists had concept's like Roko's Basilisk, a thought experiment where a future all-powerful AI might choose to punish anyone who predicted the arrival of AI but failed to contribute to the arrival of AI.

The rationalist communities took this thought experiment so seriously that the biggest rationalist forum of the time formally banned discussion of Roko's Basilisk for 5 years. They believed it was an "info hazard" because once you knew about this risk, you either had to contribute to bringing about this future AI god or you risked being punished by it in the future.

If this is new to you, you might think I'm exaggerating or making things up to make a group look bad, but this was such a core belief that it has its own Wikipedia page: https://en.wikipedia.org/wiki/Roko%27s_basilisk

The people writing AI 2027 and AI 2040 were immersed in these cultures. That's why they think that AI is the primary topic that anyone is talking about in elections, just like the Bitcoin people all thought crypto was the only relevant election topic.

The GP comment is getting attacked, but there really is a heavy religious fervor background to the AI fears in the rationalist community. The religious angle and extremism of the forums is downplayed when in crosses over into mainstream topics, but it's been there from the start.

Re: AI 2040: Plan A

#472

Post by iugtmkbdfil834 » Sat, Jul 11, 2026, 5:55 PM UTC

themgt wrote: post #339

davemp wrote: post #321

Earlier quoted context omitted.

There is a vast gulf between theoretically possible and technologically feasible. If you can’t provide a realistic path to achieve something, you’re asking people to believe in science fiction. You could tell me that a rock’s molecules are comprised of protons, neutrons, and electrons. Blood is also entirely protons, neutrons, and electrons; so theoretically, one could rearrange stone into blood. But without an actua…

What's strange to me about these comments is they're timeless. They could have been written in 2026 or 2016 or 1966. Like, afaict, for many on HN going from ELIZA->Fable 5 just didn't cause any update to priors regarding this whole philosophical question. The argument against has remained unchanged. I don't see any point in arguing about it, I just find it very strange.

I just realized that you might really be onto something. I wonder now if it is just a function of our very human inability to let go of a known construct that has served us well until now or something else. I have my own opinions, but as strange as it sounds, this may be the HN equivalent of ok boomer moment.

Re: AI 2040: Plan A

#473

Post by iugtmkbdfil834 » Sat, Jul 11, 2026, 5:58 PM UTC

defgeneric wrote: post #448

themgt wrote: post #339

Earlier quoted context omitted.

What's strange to me about these comments is they're timeless. They could have been written in 2026 or 2016 or 1966. Like, afaict, for many on HN going from ELIZA->Fable 5 just didn't cause any update to priors regarding this whole philosophical question. The argument against has remained unchanged. I don't see any point in arguing about it, I just find it very strange.

It's a form of denial. We're getting another "de-thronement of man" on the order of Copernicus and Darwin. Some get excited, others turn away in horror. Negation is the outward expression of the desire to keep human intelligence wrapped in its mystical veil. One popular idea is that these systems will asymptotically approximate human intelligence because they're trained on mostly human-written texts. Not only is that…

Can you elaborate on this? I am clearly not aware of this line of thinking and the related contradiction.

Re: AI 2040: Plan A

#474

Post by iugtmkbdfil834 » Sat, Jul 11, 2026, 6:03 PM UTC

stalfie wrote: post #378

27183 wrote: post #336

Earlier quoted context omitted.

I used the example of 1G constant acceleration space flight in another thread which got downvoted to oblivion, but I think it's a good one. That's a technology we know how to build. We just need superconducting electronics and miniaturized fusion reactors, or a ship which is built like Project Orion to use nuclear bombs for propulsion. Now write down a blueprint for superintelligence. So I've given you two impossible…

No one imagined LLMs in their current format, it was simply a result of discovering that scaling compute and tokens produced better and better results with the Transformer architecture. The inventors of the Transformer architecture were working on better translation, and probably did not imagine that their architecture would lead to modern LLMs. Imagining something in advance is not necessary at all for scientific ad…

That is simply not accurate. There are examples of scifi novels, novellas and other media that dealt with it. We can argue over whether it was that exact format, implementation and so on, but that 'shape' ( to use a common llm term ) of technological advances was very much explored.

Re: AI 2040: Plan A

#475

Post by fwipsy » Sat, Jul 11, 2026, 6:11 PM UTC

notahacker wrote: post #432

HDThoreaun wrote: post #422

Earlier quoted context omitted.

And what movement isn’t attractive to manipulative sociopaths who want to maximize their power? That’s unavoidable when dealing with humans.

An important adjacent point is the same people that are insisting that only their research can stop intelligent manipulative computers from controlling the human race are also some of the few people who believed that OpenAI was a philanthropic endeavour and that Sam Bankman Fried was trustworthy...

I'm not seeing any claims to exclusivity. I'm seeing people actively encouraging more people to enter the field of AI safety/security and openness to robust debate, etc.

I agree that there seems to be a huge problem with value drift--either people who used to research AI safety, pivoting to building AI (looking at Anthropic) or people who only ever paid lip service to safety (I tend to put Musk and Altman in this category.) These people need to be held accountable, but it doesn't mean every AI safety researcher ever was a stooge or a fraud.

Re: AI 2040: Plan A

#476

Post by Leynos » Sat, Jul 11, 2026, 6:12 PM UTC

YeGoblynQueenne wrote: post #340

taurath wrote: post #277

Earlier quoted context omitted.

Yeah I think its possible that for many folks its the first time they're coming up on these concepts, and it troubles them in the same way that the concept of death troubles them (and me, to be clear!). For me its as simple as watching how people talk, and seeing how in every single case whatever the next thing is, if you believe It, there is only ever justification of doubling down, doing more, going deeper, reducin…

>> Even if one throws that aside, spending time exploring and building with the most state of the art LLMs is just as instructive. I'm watching the implementation - whats working is ML models trained on specific domains (not much different than 5+ years ago), and whats not working is a general model that humanity can let go to work on its own. Sit in front and observe ideas turn to the samey intellectual, high-syllab…

While there are some coding focused models (composer, for example), the majority of frontier models are pitched as general purpose. The coding harnesses for Claude and GPT are even being repurposed as general purpose knowledge work harnesses.

Re: AI 2040: Plan A

#477

Post by beepbooptheory » Sat, Jul 11, 2026, 6:14 PM UTC

themgt wrote: post #391

beepbooptheory wrote: post #371

Earlier quoted context omitted.

Unpack this a little bit. Why is it strange or interesting to you? What specific priors need to be updated for us here? What is the philosophical questions at play for you?

To meta-unpack a little bit ... it is strange to me that Fable is far more capable of discussing these questions than apparently 99% of humans. Along with being more capable at quite a lot else than most humans.

OK.. when you discuss these things with your Fable, what topics come up? Can you articulate one of the questions? I am probably just another dumb human FYI, but just try it out and we will see if I can follow along.

Re: AI 2040: Plan A

#478

Post by nradov » Sat, Jul 11, 2026, 6:16 PM UTC

fwipsy wrote: post #384

romanhounds wrote: post #383

Earlier quoted context omitted.

>Given that you apparently share their goals (a better world,) isn't it worth at least hearing them out on their methods? Their methods are about convincing others that things that enrich and empower themselves at the expense of others is "improving the world". This isn't the stance of serious people who want to improve the world.

Please don't confuse AI researchers and AI safety researchers (although there is some overlap.) That's like confusing the people from the Manhattan project with the people who protested 3 Mile Island, because they're both focused on nuclear technology. One effect of EA is that every AI researcher says they're trying to make the world better. Some of them are full of shit, but not all. I'm fairly certain the authors w…

Right, the so-called "AI safety researchers" are worthless clowns and grifters with nothing of value to contribute. We would all be better off if they spent their time on something productive like writing entertaining sci-fi stories. They certainly seem to have the rich imagination necessary for that occupation.

Re: AI 2040: Plan A

#479

Post by zdragnar » Sat, Jul 11, 2026, 6:17 PM UTC

0xDEAFBEAD wrote: post #438

zdragnar wrote: post #433

Earlier quoted context omitted.

Fable 5 doesn't represent anything new, other than scale and some refinement techniques, over the original LLMs. In the chase for AGI specifically, LLMs are a dead end, just like all the other AI technologies that died in the AI winter. What priors should be updated?

>Fable 5 doesn't represent anything new, other than scale and some refinement techniques, over the original LLMs. Yet it is generating billions in revenue which Eliza did not. Perhaps all we need is scale and some refinement techniques to eat a big fraction of the economy. If unimpressive inputs lead to impressive outputs, that should make you more worried, not less.

The commercial viability is orthogonal to whether it achieves AGI, which is effectively what "reproducing the human brain" amounts to in this discussion.

Re: AI 2040: Plan A

#480

Post by notahacker » Sat, Jul 11, 2026, 6:23 PM UTC

HDThoreaun wrote: post #452

notahacker wrote: post #432

Earlier quoted context omitted.

An important adjacent point is the same people that are insisting that only their research can stop intelligent manipulative computers from controlling the human race are also some of the few people who believed that OpenAI was a philanthropic endeavour and that Sam Bankman Fried was trustworthy...

OpenAI now controls one of if not the largest philanthropic endowments on the planet. They are still a philanthropic endeavor, though I agree the non profit bait and switch as well as the Altman board situation were tragedies.

I mean, they're also gearing up for one of the largest IPOs in history to generate unprecedented amounts of wealth for themselves, and the nonprofit foundation's focus is community engagement with OpenAI products. Goldman Sachs has a large philanthropic endowment but I wouldn't confuse that with their priorities being philanthropy.

The point was I think pretty much everyone else saw the bait and switch coming...