Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Why AI companies want you to be afraid of them

bbc.com

Post reply on HN
Print view View original

91–100 of 230 posts

Re: Why AI companies want you to be afraid of them

#91

Post by californical »

Jtarii wrote: post #77
balls187 wrote: post #70

Earlier quoted context omitted.

Claude does not have a "theory" of anything, and I'd argue applying that mental model to LLM+Tools is a major reason why Claude can delete a production database.

Well, humans also routinely accidentely delete production databases. I think at this point arguing that LLMs are just clueless automatons that have no idea what they are doing is a losing battle.

I mean maybe it’s a losing battle today, but it is correct. So in a few years when the dust settles, we’ll probably all be using LLMs as clueless automatons that still do useful work as tools

Re: Why AI companies want you to be afraid of them

#92

Post by SpicyLemonZest »

b00ty4breakfast wrote: post #65
Imnimo wrote: post #31

My read is not so much "if we say this is dangerously powerful, it will make people want to buy our product", but rather that there is a significant segment of AI researchers for whom x-risk, AI alignment, etc. is a deal-breaker issue. And so the Sam Altmans of the world have to treat these concerns as serious to attract and retain talent. See for example OpenAI's pledge to dedicate 20% of their compute to safety res…

Maybe Altman specifically is only paying lip service to this stuff, but when a company like Anthropic is like "BRO MYTHOS IS TOO DANGEROUS BRO WE CANT EVEN RELEASE IT BRO JUST TRUST US BRO", my bullshit detector is beeping too loud to ignore. It's very obviously a publicity stunt, because if it were actually that dangerous you wouldn't be making such a press release, you'd be keeping your mouth shut and working to ma…

They explained in detail why they felt they had to talk about it. They think there's no safe deployment strategy other than fixing all the vulnerabilities it's likely to find, and there are too many such vulnerabilities for them to fix without getting help from a substantial number of trusted partners.

Re: Why AI companies want you to be afraid of them

#93

Post by dataviz1000 »

boh wrote: post #35

I think the big secret is that AI is just software. In the same way that a financial firm doesn't all of sudden make a bunch of money because Microsoft shipped an update to Excel, AI is inert without intention. If there's any major successes in AI output it's because a person got it to do that. Claude Code is great, but it will also wipe out a database even though it's instructed not to (I can confirm from experience…

LLM models are a distribution. Unlike a python script or turning machine, a LLM model is capable of generating any series of tokens. Developers need stop reasoning about LLM agents as deterministic and to start to think about agents in terms of Monte Carlo and Las Vegas algorithms. It isn't enough to have an agents, it also requires a cheap verifier.

If I was a Ph.D. student today, I'd probably do a thesis on cheap verifiers for LLM agents. Since LLM agents are not reliable and therefore not very useful without it, that is a trillion dollar problem.

Once a developer groks that concept, the agents stop being scary and the potential is large.

Re: Why AI companies want you to be afraid of them

#95

Post by flats »

stuaxo wrote: post #61
codingdave wrote: post #57

Earlier quoted context omitted.

Yes, and it has been said since day one of LLMs that all we need to do is keep things that way - no action without human intervention. Just like it was said that you should never grant AI direct access to change your production systems. But the stories of people who have done exactly that and had their systems damaged and deleted show that people aren't trying to even keep such basic safety nets in place. AI is getti…

Eh, these same people will attach openclaw to production systems soon and destroy their own companies.

One does not even need OpenClaw to achieve this outcome: https://x.com/lifeof_jer/status/2048103471019434248

Re: Why AI companies want you to be afraid of them

#96

Post by add-sub-mul-div »

dataviz1000 wrote: post #93
boh wrote: post #35

I think the big secret is that AI is just software. In the same way that a financial firm doesn't all of sudden make a bunch of money because Microsoft shipped an update to Excel, AI is inert without intention. If there's any major successes in AI output it's because a person got it to do that. Claude Code is great, but it will also wipe out a database even though it's instructed not to (I can confirm from experience…

LLM models are a distribution. Unlike a python script or turning machine, a LLM model is capable of generating any series of tokens. Developers need stop reasoning about LLM agents as deterministic and to start to think about agents in terms of Monte Carlo and Las Vegas algorithms. It isn't enough to have an agents, it also requires a cheap verifier. If I was a Ph.D. student today, I'd probably do a thesis on cheap v…

If you told a programmer 30 years ago that someday we'd switch from a deterministic to nondeterministic paradigm for programming computers, they'd ask if we'd put lead back in the drinking water.

Re: Why AI companies want you to be afraid of them

#97

Post by drBonkers »

dataviz1000 wrote: post #93
boh wrote: post #35

I think the big secret is that AI is just software. In the same way that a financial firm doesn't all of sudden make a bunch of money because Microsoft shipped an update to Excel, AI is inert without intention. If there's any major successes in AI output it's because a person got it to do that. Claude Code is great, but it will also wipe out a database even though it's instructed not to (I can confirm from experience…

LLM models are a distribution. Unlike a python script or turning machine, a LLM model is capable of generating any series of tokens. Developers need stop reasoning about LLM agents as deterministic and to start to think about agents in terms of Monte Carlo and Las Vegas algorithms. It isn't enough to have an agents, it also requires a cheap verifier. If I was a Ph.D. student today, I'd probably do a thesis on cheap v…

Do you have any readings you recommend to start thinking in terms of non-deterministic algorithms and cheap verifiers?

Re: Why AI companies want you to be afraid of them

#98

Post by ctoth »

simonw wrote: post #76
ctoth wrote: post #59

Earlier quoted context omitted.

> doesn't change the fact that it's software that requires human interaction to work. Have you ever seen Claude Code launch a subagent? You've used it, right? You've seen it launch a subagent to do work? You understand that that is, in fact, Claude Code running itself, right?

I don't think subagents are representative of anything particularly interesting on the "agents can run themselves" front. They're tool calls. Claude Code provides a tool that lets the model say effectively: run_in_subagent("Figure out where JWTs are created and report back") The current frontier models are all capable of "prompting themselves" in this way, but it's really just a parlor trick to help avoid burning mor…

The mechanism being simple is the interesting part. If one large complex goal can be split into subgoals and the subgoals completed without you, then you need a lot fewer humans to do a lot more work.

The OP says AI requires human interaction to work. This simply isn't true. You know yourself that as agents get more reliable you can delegate more to them, including having them launch more subagents, thereby getting more work done, with fewer and fewer humans. The unlock is the Task tool, but the power comes from the smarter and smarter models actually being able to delegate hierarchical tasks well!

Re: Why AI companies want you to be afraid of them

#99

Post by ben_w »

Micanthus wrote: post #48

> According to critics, it benefits AI companies to keep you fixated on apocalypse because it distracts from the very real damage they're already doing to the world. Am I not allowed to be concerned about _both_? I do not believe that Sam Altman and other AI company execs believe that the singularity is imminent. If they did, they wouldn't behave so recklessly. Even if they don't care about the rest of humanity, ther…

> I do not believe that Sam Altman and other AI company execs believe that the singularity is imminent. If they did, they wouldn't behave so recklessly. Even if they don't care about the rest of humanity, there's too much risk to themselves if they actually believe what they're saying.

I don't believe Zuckerberg believes in either the promise or the danger, his presentations are far too mundane. The leaked memos suggest he may simply not care about dangers, which is worse.

Altman at least seems to think an LLM can be used as an effective tool for harm and is doing more than the bare minimum to avoid AI analogies of all the accidents and disasters from the industrial age which led to us having health and safety laws, building codes, and consumer product safety laws.

Musk clearly thinks laws only exist for him to wield against others. Tries to keep active tools which cause widespread revulsion as if a freedom of speech argument is enough.

Amodei seems to actually care even when it hurts Anthropic, as evidenced by saying "no" to the US government. It could be kayfabe, Trump is famous for it after all, but as yet I have no active reason to dismiss Amodei as merely that.

Re: Why AI companies want you to be afraid of them

#100

Post by DeathArrow »

ctoth wrote: post #59
boh wrote: post #35

I think the big secret is that AI is just software. In the same way that a financial firm doesn't all of sudden make a bunch of money because Microsoft shipped an update to Excel, AI is inert without intention. If there's any major successes in AI output it's because a person got it to do that. Claude Code is great, but it will also wipe out a database even though it's instructed not to (I can confirm from experience…

> doesn't change the fact that it's software that requires human interaction to work. Have you ever seen Claude Code launch a subagent? You've used it, right? You've seen it launch a subagent to do work? You understand that that is, in fact, Claude Code running itself, right?

A one liner shell script can run itself.
Post reply on HN