Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

For AI leaders Doom is a form of hype

erkansaka.net

Post reply on HN
Print view View original

131–140 of 178 posts

Re: For AI leaders Doom is a form of hype

#131

Post by LogicFailsMe »

svara wrote: post #53
LogicFailsMe wrote: post #38

Earlier quoted context omitted.

Explain a path from today to the extermination of humanity in 2028 by the ASI and von Neumann replicators heading to the stars in the early 2030s without resorting to magical thinking or underpants gnomes level business plans. Engineering details are what matters here, not wishy washy things you once saw on Star Trek or The Outer Limits. Go.

I don't think we'd be okay with that if it took a few years longer, right? I thought the Bengio blog post posted here the other day answered that question quite well (ignoring your timeline). The argument is that in training for the capability to achieve certain goals, you may inadvertently also train for secondary instrumental goals you did not intend, such as aggressive behavior or deception. I'm actually no sure I…

Sure, unintentional outcomes, fair. But I'm asserting that things which are basically impossible today are not arriving any time soon. I 100% acknowledged the possibility and even the likelihood of rogue unintended behavior of AIs and in fact it's happening already. What I reject is the annihilation of humanity, the curing of cancer*, immortality, and von Neumann replicators heading to the stars anytime soon.

*TBF before we dismantled America's clinical trials infrastructure, the monkeys were making insane progress here: https://www.nytimes.com/2026/09/04/opinion/clinical-trials-d...

Re: For AI leaders Doom is a form of hype

#132

Post by pixl97 »

LogicFailsMe wrote: post #112
frabcus wrote: post #100

Earlier quoted context omitted.

Ha well, by definition what you want can't be provided! Of course these stories can only be technical up until the recursive self-improvement part. Then they're necessarily fantasy, as the AI is more intelligent than the story author. Unfortunately, this doesn't make that scenario impossible. However, we know now from Hugging Face that even without the later sci-fi powers, the LLMs are already capable of causing dama…

Explain how rogue FSD cars take down civilization. And I say that with the opinion that the only good scenes in The Fate of The Furious and Leave The World Behind were the ones with the rogue Teslas. But also, nothing new under the sun really: https://en.wikipedia.org/wiki/Maximum_Overdrive And so granted we could probably get a global 9/11 in deeply urban regions out of your scenario, we then bomb the datacenters an…

>during a disaster, and I've been through a few

Eh, about the biggest one I could see AI causing is the need for a black start on all 3 major US grids. Millions will die before it comes back up, well, if the US pulls out of it at all.

And in that case, thanks for setting up a nice place to live for someone far more ruthless than you. They'll snipe you, and enslave your women and children if things have gone that far downhill.

Or, we could avoid all that by making sure AI labs are properly regulated.

Re: For AI leaders Doom is a form of hype

#133

Post by SpicyLemonZest »

LogicFailsMe wrote: post #126
SpicyLemonZest wrote: post #119

Earlier quoted context omitted.

I've read Gary Marcus's critique of AI 2027. How do you grapple with the fact that superhuman coding arrived on schedule, despite his skepticism that it would? Doesn't that mean that the other predictions are also more plausible than we may think?

Superhuman coding has not happened. I work with Codex and Claude daily. They're fantastic for script coding, config issues, and simple projects. They lose the plot on bigger things to this day. Just last night, Astra decided to cheat to present the illusion of progress until I called its BS. And I was so looking forward to finally working with the AGI. Humans remain the muse, the common sense, and the manager of maki…

We understand "superhuman coding" to mean very different things, and I think it's a me problem that I can't figure out where you're coming from, because it doesn't sound like our practical experience differs. I'm going to have to think on this.

Re: For AI leaders Doom is a form of hype

#134

Post by palmotea »

LogicFailsMe wrote: post #83
frabcus wrote: post #41

Earlier quoted context omitted.

I think either "AI 2027" or "If Anyone Builds It, Everyone Dies" meet your criteria? I guess not literally, due to the dates? But they have substantive technical details.

No, they do not. They are both bullshit campfire ASI horror stories written by nontechnical sorts in the wishy washy invented field of AI safety. For example, I want someone with a proven track record in the field of robotics to explain the entire pathway from impressive 2026 Chinese robots to an entire self-assembly food chain from raw ore to von Neumann replicators that can travel to the stars and reproduce themsel…

> None of this makes any logical sense whatsoever.

It doesn't make any logical sense to set the bar as as you have, either. I mean if someone could provide the roadmap you're asking for, it's probably already too late.

And while we're talking about sci-fi, it seems fairly likely that an AI that could "kill us all" could fail to build a "entire self-assembly food chain from raw ore to von Neumann replicators." I'd be a civilizational collapse [1] without even birthing something new.

But it makes far more sense to worry about the AI x Late Capitalism crossover story or the havok that could be caused by some immature tech nerd amplified by AI, than far out sci-fi stuff.

[1] IMHO, even a genocidal plague probably wouldn't render humanity extinct, something like uncontacted tribe in a rainforest somewhere would probably survive and humanity would carry on (from square one, in a resource-depleted world).

Re: For AI leaders Doom is a form of hype

#135

Post by pixl97 »

LogicFailsMe wrote: post #104
Gregkion wrote: post #79

Earlier quoted context omitted.

Thats the problem with doom. What might not affect you, might affect millions across the ocean. A society no longer creating bullshit jobs because there is no reason for it anymore. You have some AI. Right now a whole industry is transforming (again) but now filling the automation gap we were unable to do because of a shortage of labor or expertise. Our society already has peak employmet reached, now AI takes away al…

Actually, some sorts do assert humanity will be wiped off the planet by AI. I don't share that view either. I think it's ludicrous. We've lived through much worse without any sort of technology beyond fire and a big stick. But the end of scarcity and disease is a fantastic thing if it happens. If our lizard brains do not bridge that transition and instead attempt to maintain the billionaire status quo, I agree, we're…

>We've lived through much worse without any sort of technology beyond fire and a big stick.

No 'we' didn't. A whole shitload of much smaller isolated communities did. The world added 7+ billion more people since then. Imagine chlorine doesn't show up tomorrow? How many people die in two weeks? How far do you travel when no gas comes out of the pumps. How much will you stand out at night if you're the only ones with power? How fast do working antibiotics run out?

The billionaires you're talking about are mostly ignorant fucking idiots with a god complex. The moment any complex system starts falling apart they are saying as dumb of shit as everyone else (during covid for example).

Most of the pasts anti-fragility is gone. We survive by means of very complex systems and if you lose too many key points in those systems they can totally collapse.

Re: For AI leaders Doom is a form of hype

#136

Post by jackb4040 »

pixl97 wrote: post #128
jackb4040 wrote: post #121

Earlier quoted context omitted.

https://archive.ph/BzBjZ Read the article. The things the Chinese officials are worried about are: * Fake videos / disinformation * Cyberattacks on critical infrastructure * Data leaks * Advances in military sensing / targeting technology These concerns have little in common with the Yudkowskian / apocalyptic doom marketing the US hyperscalers are pushing now. The Times even says this explicitly in the second paragra…

The thing is all the above are threats. The internet being filled with more trash than ever is a threat to our sanity. The anti-social posts mass generated by AI is a threat to our societies. Bullshit news stories is a threat to our political processes and democracy. Smart cameras every are a threat to our privacy and freedom. AI enabled scams are a threat to our wallets. AI hacking is a threat to anything connected…

Real threats that are happening to regular people already, vs marketing "threats" that the technology is too powerful / valuable.

The fact that you are chiming in to say you don't understand the difference between these 2 distinct categories of threats is fine, we can work on that. But the person who posted the article originally not only acknowledges the distinction, they falsely presented the article as evidence of non-western support for the latter category, even though both the Chinese official and the western reporter directly contradict that claim immediately at the beginning of the article.

Re: For AI leaders Doom is a form of hype

#137

Post by LogicFailsMe »

pixl97 wrote: post #132
LogicFailsMe wrote: post #112

Earlier quoted context omitted.

Explain how rogue FSD cars take down civilization. And I say that with the opinion that the only good scenes in The Fate of The Furious and Leave The World Behind were the ones with the rogue Teslas. But also, nothing new under the sun really: https://en.wikipedia.org/wiki/Maximum_Overdrive And so granted we could probably get a global 9/11 in deeply urban regions out of your scenario, we then bomb the datacenters an…

>during a disaster, and I've been through a few Eh, about the biggest one I could see AI causing is the need for a black start on all 3 major US grids. Millions will die before it comes back up, well, if the US pulls out of it at all. And in that case, thanks for setting up a nice place to live for someone far more ruthless than you. They'll snipe you, and enslave your women and children if things have gone that far…

Such an individual would get a quick corrective lesson in the second amendment from my friends and neighbors. But once again, hyperbole. The US grid goes down. That sucks for up to a year and a half.

But hospitals and other critical organizations have generators and the national guard would be deployed to keep them fueled. And yes it will crash the economy, which is kind of a myth anyway. Buy solar panels and batteries, live off a well, and get to know your neighbors if you think this is likely. Also consider being like the Mormons and storing a 6-12 month supply of nonperishable food.

Regulating AI achieves nothing but to hand the 21st century to China. Regulating its deployment OTOH will not be allowed by companies like Meta and Palantir whose gross margin is reliant on it. I'm favor of regulating deployment. The research not so much.

Re: For AI leaders Doom is a form of hype

#138

Post by pixl97 »

cmrdporcupine wrote: post #103
frabcus wrote: post #28

Earlier quoted context omitted.

If it helps, this person from OpenAI explains at least something: "I realize that no one has properly explained yet what all the lab employees have seen that scared them so suddenly." https://x.com/MajmudarAdam/status/2098881885200081234 The explanation is basically they have other dimensions (not just pretraining and inference compute) that scale, and they've got fairly convincing scaling laws. And they know they ca…

That posts seemingly tries to refute the idea of Dario etc's proclamations really being about regulatory capture by saying: "No, really, the engineers are just terrified!" But both things can be true at once: 1. Engineers inside these labs might genuinely be anxious or paranoid about what they are building. 2. ... at the corporate level, calling for heavy regulation, safety pauses, removal/suspension of anti-collusio…

> They are executing loss functions and reward systems and are responding to prompts.

And because we trained them on the entirety of human knowledge this happens to look like self-preservation, fear of death, and instrumental goals.

I mean, we train them in an evolutionary manner. Instrumental convergence will never happen, right?

Re: For AI leaders Doom is a form of hype

#139

Post by ahf8Aithaex7Nai »

reducesuffering wrote: post #86
ianm218 wrote: post #25

What I don't really understand with these posts is that they don't at any point contend with the "doomers" argument. Like people get so caught in people acting hypocritical/ having mixed incentives/ or just in many cases personally disliking West Coast billionaire types and the kind of employees they hire. All of which seems fair to point out. But like the core argument - AI is getting better at core research tasks (…

Because they can't. If they actually engaged with it, they would realize how shallow the "doomer" / existential risk dismissals are. Meanwhile there are ample essays and arguments about the existential issues from Turing award winners, frontier lab CEO's, and many more. There's also https://ai-2040.com/ , you will never see the equivalent on the other side of the debate because it won't hold under any scrutiny.

Can you refute the claim that the Second Coming of Christ is imminent? It’s very convenient to put forward something completely absurd and hypothetical and then expect the other side to refute it. You can have an LLM of your choice compile a list of prominent figures who hold the opposing view. The list is not empty. By the way, AI-2040 used to be AI-2027. I’m already looking forward to more updates.

Re: For AI leaders Doom is a form of hype

#140

Post by LogicFailsMe »

SpicyLemonZest wrote: post #133
LogicFailsMe wrote: post #126

Earlier quoted context omitted.

Superhuman coding has not happened. I work with Codex and Claude daily. They're fantastic for script coding, config issues, and simple projects. They lose the plot on bigger things to this day. Just last night, Astra decided to cheat to present the illusion of progress until I called its BS. And I was so looking forward to finally working with the AGI. Humans remain the muse, the common sense, and the manager of maki…

We understand "superhuman coding" to mean very different things, and I think it's a me problem that I can't figure out where you're coming from, because it doesn't sound like our practical experience differs. I'm going to have to think on this.

Define it. I haven't seen anything yet from a coding agent I couldn't write myself and usually write better. But I don't have to gold plate every line of code, just the 5% or so that eats 90+% of the cycles. And this is where the agents are weakest. This does get to one of the reasons I don't believe AI cures death or builds von Neumann replicators any time soon because that requires out of sample breakthroughs. And that's not their strong point.

Just like AlphaFold 2 genuinely increased the weak sequence but strong structural homology detection of proteins, but it did far less for out of sample domains and it provides no intel into the kinetics of folding. Details matter. And nobody likes to hear about them.

Post reply on HN