Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Why AI companies want you to be afraid of them

bbc.com

Post reply on HN
Print view View original

131–140 of 230 posts

Re: Why AI companies want you to be afraid of them

#131

Post by munk-a »

recursive wrote: post #79
fnoef wrote: post #64

Earlier quoted context omitted.

My Linux server runs a cron job, that can spin off a thread and even use other ~apps~ tools. Did I invent AGI?

Maybe. But probably not. It doesn't matter if it's AGI though. If those other apps and tools do simple things that are predictable, then we can be pretty sure what will happen. If those tools can modify their own configuration and create new cron jobs, it becomes much harder to say anything about what will happen.

Most of us work on software that can modify its own configuration and create new jobs. I, too, have worked in ansible and terraform.

The key break here is the lack of predictability and I think it's important that we don't get too starry eyed and accept that that might be a weakness - not a strength.

Re: Why AI companies want you to be afraid of them

#132

Post by xboxnolifes »

ctoth wrote: post #59
boh wrote: post #35

I think the big secret is that AI is just software. In the same way that a financial firm doesn't all of sudden make a bunch of money because Microsoft shipped an update to Excel, AI is inert without intention. If there's any major successes in AI output it's because a person got it to do that. Claude Code is great, but it will also wipe out a database even though it's instructed not to (I can confirm from experience…

> doesn't change the fact that it's software that requires human interaction to work. Have you ever seen Claude Code launch a subagent? You've used it, right? You've seen it launch a subagent to do work? You understand that that is, in fact, Claude Code running itself, right?

My claude has never yet launched itself from my terminal, gave itself a prompt, and then got to work. It has only ever spawned a sub-agent after I had given it a prompt. It was inert until a human got involved.

If that is software running itself, then an if statement that spawns a process conditionally is running itself.

Re: Why AI companies want you to be afraid of them

#133

Post by bellBivDinesh »

boh wrote: post #35

I think the big secret is that AI is just software. In the same way that a financial firm doesn't all of sudden make a bunch of money because Microsoft shipped an update to Excel, AI is inert without intention. If there's any major successes in AI output it's because a person got it to do that. Claude Code is great, but it will also wipe out a database even though it's instructed not to (I can confirm from experience…

The specter of AGI helps them obfuscate this

Re: Why AI companies want you to be afraid of them

#134

Post by AndrewKemendo »

ben_w wrote: post #123
AndrewKemendo wrote: post #107

Earlier quoted context omitted.

Sounds like a pretty efficient self correcting mechanism I’m not sure what the problem is there

Normalisation of deviance is the problem: https://en.wikipedia.org/wiki/Normalization_of_deviance Remember that these models are getting better; this means they get trusted with increasingly more important things by the time an error explodes in someone's face. It would be very bad if the thing which explodes is something you value which was handed off to an AI by someone who incorrectly thought it safe. AI companies…

That’s not a technical/AI problem in any sense, that’s a social problem in organizing and coordinating control structures

Further, it’s only a problem to the extent that the downsides or risks are not accounted for which again… is a social problem not a technological problem

This isn’t a problem for organizations that have well aligned incentives across their workflows

A well organized company that has solid incentives is not going to diminish their own capacity by prematurely deploying a technology that is not capable of actually improving

The issue is that 99% of the organizations that people deal with have entirely orthogonal incentives to them. They are then attributing the pain in dealing with that organization to the technology rather than the misaligned incentives

Re: Why AI companies want you to be afraid of them

#135

Post by goatlover »

georgemcbay wrote: post #121
add-sub-mul-div wrote: post #96

Earlier quoted context omitted.

If you told a programmer 30 years ago that someday we'd switch from a deterministic to nondeterministic paradigm for programming computers, they'd ask if we'd put lead back in the drinking water.

> they'd ask if we'd put lead back in the drinking water. With Lee Zeldin heading the EPA is anyone sure we won't?

Replace fluoride with lead in the water. Blocks out all the negative effects from wind turbines. /s

Re: Why AI companies want you to be afraid of them

#136

Post by Tangurena2 »

bryan0 wrote: post #25

> Why do AI companies want us to be afraid of them? ... According to critics, it benefits AI companies to keep you fixated on apocalypse because it distracts from the very real damage they're already doing to the world. People seem unable to make up their mind if AI is very dangerous or is it not. I think what the AI companies and this author agree on, is that this technology is potentially extremely dangerous. AI im…

> People seem unable to make up their mind if AI is very dangerous or is it not.

This is a propaganda tactic. For decades, tobacco companies claimed that there was no evidence that smoking was bad for one's health. Then, only after losing dozens of lawsuits did the propaganda switch to "but everyone knew for 100+ years that smoking was lethal".

One can read about it by reading Trust Us, We're Experts, or Toxic Sludge Is Good For You, or the other books written by the authors.

https://en.wikipedia.org/wiki/Trust_Us,_We%27re_Experts

https://www.prwatch.org/tsigfy.html

Re: Why AI companies want you to be afraid of them

#137

Post by lbrito »

deepsquirrelnet wrote: post #50

This is my own take, directly related to this that I posted a little while back. The one thing that I think the article missed is the geopolitical angle they’re also working: * We need to completely deregulate these US companies so China doesn't win and take us over * We need to heavily regulate anybody who is not following the rules that make us the de-facto winner * This is so powerful it will take all the jobs (an…

>deregulation

Peter Thiel literally gave a lecture on the Antichrist* saying basically that regulation is satanic https://www.nytimes.com/2026/03/17/world/europe/peter-thiel-...

* He's the best person in the world for this lecture - the only one that can claim first-person knowledge on the subject!

Re: Why AI companies want you to be afraid of them

#138

Post by adrian_b »

jrumbut wrote: post #56
FiberBundle wrote: post #32

Another potential reason, not mentioned in the article, is that open source models obviously pose the biggest threat in the labs' ability to monetize their tech. Anthropic especially seems to be very anti open-source. If frontier models start to plateau and don't have capabilities that truly differentiate them, nobody will pay what the labs would want to charge. Posing the tech as a danger is a way for them to make t…

This is a great point. I'm kind of surprised there isn't a greater proliferation of open source models to do things the public ones won't. I know such things exist, but imagine how many web browsers there would be if all the mainstream ones had the same content restrictions as LLMs. I guess since training them does take cash that raises the bar for what people will do as a prank or on principle.

Training is time-consuming and/or expensive but it is not the main blocker.

The main problem is obtaining a big enough training data set. Now, unless you are someone like Google or Microsoft, it has become much harder to scrap data from the Internet than by the time when OpenAI and Anthropic got most of their data.

Re: Why AI companies want you to be afraid of them

#139

Post by Terr_ »

codingdave wrote: post #57
boh wrote: post #35

I think the big secret is that AI is just software. In the same way that a financial firm doesn't all of sudden make a bunch of money because Microsoft shipped an update to Excel, AI is inert without intention. If there's any major successes in AI output it's because a person got it to do that. Claude Code is great, but it will also wipe out a database even though it's instructed not to (I can confirm from experience…

Yes, and it has been said since day one of LLMs that all we need to do is keep things that way - no action without human intervention. Just like it was said that you should never grant AI direct access to change your production systems. But the stories of people who have done exactly that and had their systems damaged and deleted show that people aren't trying to even keep such basic safety nets in place. AI is getti…

My cynical rule of thumb: By default we should imagine LLMs like javascript logic offloaded into a stranger's web-browser.

The risks are similar: No prompts/data that go in can reliably be kept secret; A sufficiently-motivated stranger can have it send back completely arbitrary results; Some of those results may trigger very bad things depending on how you use or even just display them on your own end.

P.S. This conceptual shortcut doesn't quite capture the dangers of poison data, which could sabotage all instances even when they happen to be hosted by honorable strangers.

Re: Why AI companies want you to be afraid of them

#140

Post by goatlover »

chis wrote: post #37
Imnimo wrote: post #31

My read is not so much "if we say this is dangerously powerful, it will make people want to buy our product", but rather that there is a significant segment of AI researchers for whom x-risk, AI alignment, etc. is a deal-breaker issue. And so the Sam Altmans of the world have to treat these concerns as serious to attract and retain talent. See for example OpenAI's pledge to dedicate 20% of their compute to safety res…

Yeah I just don't buy that it would somehow help AI companies for everyone to be existentially afraid of their technology. It seems much more reasonable to think that they really believe the things they're saying, than that it's some kind of 4d chess. Additionally Dario has just been really accurate with his predictions so far. For instance in early 2025 he predicted that nearly 100% of code would be written with AI…

I'd argue if they really believed AI was an existential threat, they would shut down research and encourage everyone else to halt R&D. But then again, the Cold War happened, even over the objections of physicists like Einstein & Oppenheimer.
Post reply on HN