Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Why AI companies want you to be afraid of them

bbc.com

Post reply on HN
Print view View original

71–80 of 230 posts

Re: Why AI companies want you to be afraid of them

#71

Post by mofeien »

deepsquirrelnet wrote: post #50

This is my own take, directly related to this that I posted a little while back. The one thing that I think the article missed is the geopolitical angle they’re also working: * We need to completely deregulate these US companies so China doesn't win and take us over * We need to heavily regulate anybody who is not following the rules that make us the de-facto winner * This is so powerful it will take all the jobs (an…

"The race to build smarter-than-human AI is a race with no winners."

And specifically about the point on China, several people in power in China have also expressed the need to regulate AI and put international structures of governance in place to make sure it will benefit mankind:

https://nowinners.ai/#s5-china

Re: Why AI companies want you to be afraid of them

#72

Post by b00ty4breakfast »

chis wrote: post #37
Imnimo wrote: post #31

My read is not so much "if we say this is dangerously powerful, it will make people want to buy our product", but rather that there is a significant segment of AI researchers for whom x-risk, AI alignment, etc. is a deal-breaker issue. And so the Sam Altmans of the world have to treat these concerns as serious to attract and retain talent. See for example OpenAI's pledge to dedicate 20% of their compute to safety res…

Yeah I just don't buy that it would somehow help AI companies for everyone to be existentially afraid of their technology. It seems much more reasonable to think that they really believe the things they're saying, than that it's some kind of 4d chess. Additionally Dario has just been really accurate with his predictions so far. For instance in early 2025 he predicted that nearly 100% of code would be written with AI…

it pushes the idea that these programs are super amazing and powerful to people who are non-technical. It also allows them to control the narrative of how exactly AI is dangerous to society. Rather than worry about the energy consumption of all these new datacenters, they can redirect attention to some far-off concern about SHODAN taking over Citadel Station and turning the inhabitants into cyber-mutants or whatever.

Re: Why AI companies want you to be afraid of them

#73

Post by parliament32 »

ctoth wrote: post #66
fnoef wrote: post #64

Earlier quoted context omitted.

My Linux server runs a cron job, that can spin off a thread and even use other ~apps~ tools. Did I invent AGI?

Does your Linux server decide what processes it should launch at what time with a theory of what will happen next in order to complete a goal you specified in natural language? If so yes, I reckon you sure have!

So... systemd is AGI now?

Re: Why AI companies want you to be afraid of them

#74

Post by mofeien »

scratchyone wrote: post #2

Honestly we should have learned this claim from AI companies was purely fear-mongering back when GPT-2 was "too dangerous to release".

Given that his reason for saying GPT-2 was too dangerous to release was that the world needed more time to prepare for the effects of this technology, and given that the following models were basically scaled-up versions of it and killed social media, news reporting and other kinds of communication, I'd say he was right about the dangers of it.

Re: Why AI companies want you to be afraid of them

#75

Post by cyanydeez »

boh wrote: post #35

I think the big secret is that AI is just software. In the same way that a financial firm doesn't all of sudden make a bunch of money because Microsoft shipped an update to Excel, AI is inert without intention. If there's any major successes in AI output it's because a person got it to do that. Claude Code is great, but it will also wipe out a database even though it's instructed not to (I can confirm from experience…

I think the market isn't for anyone but other businesses. We're all ants trying to understand how AI is going to eradicate the lower levels of society.

Re: Why AI companies want you to be afraid of them

#76

Post by simonw »

ctoth wrote: post #59
boh wrote: post #35

I think the big secret is that AI is just software. In the same way that a financial firm doesn't all of sudden make a bunch of money because Microsoft shipped an update to Excel, AI is inert without intention. If there's any major successes in AI output it's because a person got it to do that. Claude Code is great, but it will also wipe out a database even though it's instructed not to (I can confirm from experience…

> doesn't change the fact that it's software that requires human interaction to work. Have you ever seen Claude Code launch a subagent? You've used it, right? You've seen it launch a subagent to do work? You understand that that is, in fact, Claude Code running itself, right?

I don't think subagents are representative of anything particularly interesting on the "agents can run themselves" front.

They're tool calls. Claude Code provides a tool that lets the model say effectively:

  run_in_subagent("Figure out where JWTs are created and report back")
The current frontier models are all capable of "prompting themselves" in this way, but it's really just a parlor trick to help avoid burning more tokens in the top context window.

It's a really useful parlor trick, but I don't think it tells us anything profound.

Re: Why AI companies want you to be afraid of them

#77

Post by Jtarii »

balls187 wrote: post #70
ctoth wrote: post #66

Earlier quoted context omitted.

Does your Linux server decide what processes it should launch at what time with a theory of what will happen next in order to complete a goal you specified in natural language? If so yes, I reckon you sure have!

Claude does not have a "theory" of anything, and I'd argue applying that mental model to LLM+Tools is a major reason why Claude can delete a production database.

Well, humans also routinely accidentely delete production databases. I think at this point arguing that LLMs are just clueless automatons that have no idea what they are doing is a losing battle.

Re: Why AI companies want you to be afraid of them

#78

Post by scottyah »

mbesto wrote: post #52
edbaskerville wrote: post #43

Earlier quoted context omitted.

Does anyone have good estimates of what percent of real production code is currently being written by LLMs? (& presumably this is rather different for your typical SaaS backend vs. frontend vs. device drivers vs. kernel schedulers...)

By all companies? I'd say less than 10% of all LOC today are generated by LLMs.

Really? In my bubble of internet news it seems the sheer number of companies that have formed and shipped LLM code to production has already surpassed existing companies. I've personally shipped dozens of (mediocre) human months or years worth of code to "production", almost certainly more than I've ever done for companies I've worked at (to be fair I've been a lot more on the SRE side for a few years now).

Re: Why AI companies want you to be afraid of them

#79

Post by recursive »

fnoef wrote: post #64
ctoth wrote: post #59

Earlier quoted context omitted.

> doesn't change the fact that it's software that requires human interaction to work. Have you ever seen Claude Code launch a subagent? You've used it, right? You've seen it launch a subagent to do work? You understand that that is, in fact, Claude Code running itself, right?

My Linux server runs a cron job, that can spin off a thread and even use other ~apps~ tools. Did I invent AGI?

Maybe. But probably not. It doesn't matter if it's AGI though. If those other apps and tools do simple things that are predictable, then we can be pretty sure what will happen. If those tools can modify their own configuration and create new cron jobs, it becomes much harder to say anything about what will happen.

Re: Why AI companies want you to be afraid of them

#80

Post by dodu_ »

bryan0 wrote: post #25

> Why do AI companies want us to be afraid of them? ... According to critics, it benefits AI companies to keep you fixated on apocalypse because it distracts from the very real damage they're already doing to the world. People seem unable to make up their mind if AI is very dangerous or is it not. I think what the AI companies and this author agree on, is that this technology is potentially extremely dangerous. AI im…

>So maybe instead of demonizing the AI companies, we have a nuanced debate about this tech and propose solutions that our best for our society?

These are not mutually exclusive.

Calling out the demonic behavior of trying to coerce people into using your product out of fear is not an indictment of the underlying technology itself.

Post reply on HN