Live data from Hacker News

AI 2040 and the cult of intelligence

geohot.github.io

181–190 of 280 posts

Re: AI 2040 and the cult of intelligence

#181
post #36

Earlier quoted context omitted.

Does it matter in 2026?

In constitutional / jurisdictional / legal terms, perhaps not. In forming consensus about the philosophical underpinnings of the rights we have with respect to the emerging internet? I think so.

How can some words written nearly 250 years ago by some rich slave owners be of any relevance to a modern world

Re: AI 2040 and the cult of intelligence

#182

Yep, that's horrifying. Imagine LLMs not just denying information about, to intentionally choose a hot button example, abortions but also invisibly logging a black mark against the asker for thoughtcrime/precrime because the current ruling party has baked their ideology into law. [EDIT] Imagine subtle injection of biases into responses that support the agenda of the current ruling party and failing to include any cou…

I don't know what's to imagine here, grok has already been found doing it and people continue to use it.

Re: AI 2040 and the cult of intelligence

#183
I am, mostly, aligned with the author here, though I don't think that talking to AI about how to not get caught for killing your wife is really going all the way in.

Going all the way in is talking to it about planning a mass shooting. Or producing a nuclear device or bioweapon. Sure, the knowledge itself isn't dangerous unless combined with other traditional behaviors like acquiring certain materials, but maybe there should be a line. Living free shouldn't be a suicide pact.

Re: AI 2040 and the cult of intelligence

#184
post #124
post #92

Earlier quoted context omitted.

Has AI yet been a serious problem?

Yes, repeatedly, since well before this particular AI summer. One previous attempt at AI was "expert systems", and while the term fell out of use it's functionally about the same as basically all modern business systems, and doing that wrong led to a lot of people being prosecuted for crimes they didn't commit: https://en.wikipedia.org/wiki/British_Post_Office_scandal If you insist on learning systems, all AB testing…

Like, is all of statistics just basically AI, or is there a meaningful difference?

Re: AI 2040 and the cult of intelligence

#185

Earlier quoted context omitted.

And our robots still look far from capable of changing bike tires...

That's not OP's contention though; it's that models are incapable of even surfacing sufficient information. This doesn't pass the smell test. To wit, here is ChatGPT's "superintelligent" answer: https://chatgpt.com/s/t_6a52b4a322bc81919308ec864a0cc056 Seems like a perfectly cromulent description by which a beginner could probably change a tire, perhaps with a few follow-ups to understand what certain components are.

I read his contention as “the people who wrote AI 2040 are nerds who are so ungrounded from physical reality they wouldn’t even know how to change a bike tire if given instructions”. Which I think is also wrong but has a genuine criticism in it.

Re: AI 2040 and the cult of intelligence

#186
post #178

Earlier quoted context omitted.

He made specific predictions about what problems AI would cause, not just a vague "AI will be dangerous". Those specific predictions were entirely incorrect; they both said that things would happen that didn't, and failed to foresee any of the actual problems with AI that have happened.

Yudkowsky said AI would be a problem when very few people taken seriously in the mainstream were saying it was going to be a problem. Saying "specific predictions are wrong" is missing the forest for the trees here.

To strain the metaphor, there's no trees! Yudkowsky predicted that there'd be a forest but it's actually grassland and you're saying that he's right because there's plants. AI has not developed in even slightly the way he claimed it would develop, none of the problems he said AI would cause have happened, and a bunch of things he never even thought of have happened.

People pushed back against his AI doom claims not because the idea that AI could be bad was unthinkable (just look at literally every SF story from the last 50 years featuring AI?), but because he specifically was worried about AI becoming God. That is no closer to happening than it was 30 years ago.

Re: AI 2040 and the cult of intelligence

#187
post #124
post #92

Earlier quoted context omitted.

Has AI yet been a serious problem?

Yes, repeatedly, since well before this particular AI summer. One previous attempt at AI was "expert systems", and while the term fell out of use it's functionally about the same as basically all modern business systems, and doing that wrong led to a lot of people being prosecuted for crimes they didn't commit: https://en.wikipedia.org/wiki/British_Post_Office_scandal If you insist on learning systems, all AB testing…

Horizon was accounting software - where is the connection to expert systems?

Sounds like corporations can have bad motivations, and can make bad software - no AI required.

Re: AI 2040 and the cult of intelligence

#188

Yep, that's horrifying. Imagine LLMs not just denying information about, to intentionally choose a hot button example, abortions but also invisibly logging a black mark against the asker for thoughtcrime/precrime because the current ruling party has baked their ideology into law. [EDIT] Imagine subtle injection of biases into responses that support the agenda of the current ruling party and failing to include any cou…

Rumor is that you do in fact get a black mark [0]… or, conversely, a “vetted” mark [1]… tied back certainly to your payment method and increasingly to your government name [2].

[0] e.g. https://docs.bswen.com/blog/2026-03-21-detect-claude-hidden-... , https://www.anthropic.com/news/building-safeguards-for-claud... , https://www.techtimes.com/articles/319415/20260701/claude-co...

[1] e.g. https://support.claude.com/en/articles/14604842-real-time-cy...

[2] https://support.claude.com/en/articles/14328960-identity-ver...

Re: AI 2040 and the cult of intelligence

#189

Earlier quoted context omitted.

I would have been very hesitant to run the “just killed wife“ test given that ChatGPT will indeed flag your account and if a certain threshold is crossed, escalate to humans, who will presumably contact authorities to avoid this happening again: https://www.bbc.com/news/articles/cq6je7e80r7o Obviously a certain percentage of the user base is running lurid hypotheticals through the system all the time, but I don’t dou…

> ChatGPT will indeed flag your account Oh no, I'll have to spin up a new e-mail address...

And payment method, presumably. At least until they get the KYC stuff revved up…

https://support.claude.com/en/articles/14328960-identity-ver...

> Being responsible with powerful technology starts with knowing who is using it.

Shiver.

Re: AI 2040 and the cult of intelligence

#190
post #178

Earlier quoted context omitted.

Yudkowsky said AI would be a problem when very few people taken seriously in the mainstream were saying it was going to be a problem. Saying "specific predictions are wrong" is missing the forest for the trees here.

To strain the metaphor, there's no trees! Yudkowsky predicted that there'd be a forest but it's actually grassland and you're saying that he's right because there's plants. AI has not developed in even slightly the way he claimed it would develop, none of the problems he said AI would cause have happened, and a bunch of things he never even thought of have happened. People pushed back against his AI doom claims not b…

I mean I think I agree with most of what you’re saying, I do agree it’s a bit of a forest grasslands scenario, but the key difference is that he said there’d be a forest at a time when you’d be hard pressed to find someone saying there would be even a microorganism on the ground. It’s pretty darn easy today in 2026 to say wow, his predictions were so off. And it is true: they are very wrong compared to any other prediction made in the last 5 years. But few people were saying this stuff in 2008.
Post reply on HN