Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

For AI leaders Doom is a form of hype

erkansaka.net

Post reply on HN
Print view View original

41–50 of 177 posts

Re: For AI leaders Doom is a form of hype

#41

Post by frabcus »

LogicFailsMe wrote: post #38
palmotea wrote: post #30

Earlier quoted context omitted.

> The timing of the AI doom narrative could not come as a more suspect time....There's little hope investors could recoup the wild investments made into Anthropic and OpenAI Personally, I think we should take the doom narratives very seriously, and act on them, but make sure those actions are not in the interests of Anthropic and OpenAI investors.

Explain a path from today to the extermination of humanity in 2028 by the ASI and von Neumann replicators heading to the stars in the early 2030s without resorting to magical thinking or underpants gnomes level business plans. Engineering details are what matters here, not wishy washy things you once saw on Star Trek or The Outer Limits. Go.

I think either "AI 2027" or "If Anyone Builds It, Everyone Dies" meet your criteria?

I guess not literally, due to the dates? But they have substantive technical details.

Re: For AI leaders Doom is a form of hype

#42

Post by atleastoptimal »

This article reads like it was written by Claude.

Also those at Anthropic genuinely believe AI could lead to doom, that's why they started the company, they believe only they can be stewards of digital gods. Whether you believe it or not, they genuinely do, and wouldn't voluntarily sow fear, uncertainty and doubt in their product when it doesn't guarantee benefits.

This is an instance of work in a genre I call "AI lab grievance poetry" where a vast collection of criticisms are levied against AI labs, with specifc ad-hominems towards the executives, while avoiding the core of their arguments about AI risk

Re: For AI leaders Doom is a form of hype

#44

Post by orbital-decay »

frabcus wrote: post #41
LogicFailsMe wrote: post #38

Earlier quoted context omitted.

Explain a path from today to the extermination of humanity in 2028 by the ASI and von Neumann replicators heading to the stars in the early 2030s without resorting to magical thinking or underpants gnomes level business plans. Engineering details are what matters here, not wishy washy things you once saw on Star Trek or The Outer Limits. Go.

I think either "AI 2027" or "If Anyone Builds It, Everyone Dies" meet your criteria? I guess not literally, due to the dates? But they have substantive technical details.

No, and neither does Star Trek.

Re: For AI leaders Doom is a form of hype

#46

Post by shimman »

palmotea wrote: post #30
mrinterweb wrote: post #24

The timing of the AI doom narrative could not come as a more suspect time. Chinese labs are releasing open models that are nipping at the heals of the frontier models. Anthropic and OpenAI know these frontier class open weight models can threaten their valuations by changing inference to a commodity where the provider who can serve the best models the cheapest and fastest will win. Few are going to pay $50/million to…

> The timing of the AI doom narrative could not come as a more suspect time....There's little hope investors could recoup the wild investments made into Anthropic and OpenAI Personally, I think we should take the doom narratives very seriously, and act on them, but make sure those actions are not in the interests of Anthropic and OpenAI investors.

[flagged]

Re: For AI leaders Doom is a form of hype

#47

Post by pizza234 »

> and it concerns containment failure, reward hacking, and evaluation integrity, a governance and engineering problem, rather than machine malevolence.

The doom scenario is more nuanced and technical than "Evil AI", for starters:

- "Evil AI" is not what is predicted, which instead is the much more banal "humans are impediment to AI's tasks, let's remove them (part or whole)"; to understand this, just imagine the relationship between humans and ants

- "reward hacking, and evaluation integrity": this is way more problematic (and most importantly, architectural) than it sounds: AIs are becoming more and more opaque; it may be impossible to inspect a model's alignment (unless a lot of research in poured on this topic, which is another aspect of today's pacing problem) and it may be impossible to understand if a model is truly aligned or it's faking

Re: For AI leaders Doom is a form of hype

#48

Post by throwitaway222 »

palmotea wrote: post #30
mrinterweb wrote: post #24

The timing of the AI doom narrative could not come as a more suspect time. Chinese labs are releasing open models that are nipping at the heals of the frontier models. Anthropic and OpenAI know these frontier class open weight models can threaten their valuations by changing inference to a commodity where the provider who can serve the best models the cheapest and fastest will win. Few are going to pay $50/million to…

> The timing of the AI doom narrative could not come as a more suspect time....There's little hope investors could recoup the wild investments made into Anthropic and OpenAI Personally, I think we should take the doom narratives very seriously, and act on them, but make sure those actions are not in the interests of Anthropic and OpenAI investors.

So if the end of the world can be stopped, but the only specific action to do so helps Anthropic/OpenAI (and it's proven) you don't pull that lever?

Re: For AI leaders Doom is a form of hype

#49

Post by phoghed »

NoahZuniga wrote: post #19

> I do not doubt that these people are sincere in some private, psychological sense ... . As a media scholar, what interests me is not whether Hubinger believes his own number. What interests me is what this genre of utterance does in the world ... . My complaint is not that these statements are alarming. It is that they are apocalyptic in form, hypocritical in structure, and strategically productive in effect — and…

Why not? Why can’t the believe their statements and also use them to generate hype?

If they sincerely believed them they shouldn’t be creating the doom machine to begin with anyway. If they believe it and keep going, are they even worth listening to on this topic? Beyond of course taking it at face value “I’m creating this doomsday machine and I will kill you with it inadvertently because it’s earning me a fuck ton of money and power”

Re: For AI leaders Doom is a form of hype

#50

Post by swingboy »

frabcus wrote: post #28
swingboy wrote: post #6

Can anyone from a lab anonymously confirm they have or are insanely close to real-time weight modifications and still using the same architecture every other lab is using (transformers)?

If it helps, this person from OpenAI explains at least something: "I realize that no one has properly explained yet what all the lab employees have seen that scared them so suddenly." https://x.com/MajmudarAdam/status/2098881885200081234 The explanation is basically they have other dimensions (not just pretraining and inference compute) that scale, and they've got fairly convincing scaling laws. And they know they ca…

I’m apprehensive about using the recent hacking examples as proof that we’re “not even close to the wall” simply because such scenarios haven’t happened before. There’s a big difference between agents eventually hacking something because they just don’t get tired and can essentially brute force their way to a goal and super-intelligence. To be clear, I’m not saying the Hugging Face or Navier Stokes incidents aren’t impressive.
Post reply on HN