Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Anthropic’s paper smells like bullshit

djnn.sh

Post reply on HN
Print view View original

51–60 of 349 posts

Re: Anthropic’s paper smells like bullshit

#51

Post by padolsey »

EMM_386 wrote: post #43
padolsey wrote: post #30

Earlier quoted context omitted.

> What would the IoCs even be? Prompts.

The prompts aren't the key to the attack, though. They were able to get around guardrails with task decomposition. There is no way for the AI system to verify whether you are white hat or black hat when you are doing pen-testing if the only task is to pen-test. Since this is not part of a "broader attack" (in the context), there is no "threat". I don't see how this can be avoided, given that there are legitime uses t…

I agree that there will be no single call or inference that presents malice. But I feel like they could still share general patterns of orchestration (latencies, concurrencies, general cadences and parallelization of attacks, prompts used to granulaize work, whether prompts themselves have been generated in previous calls to Claude). There's a bunch of more specific telltales they could have alluded to. I think it's likely they're being obscure because they don't want to empower bad actors, but that's not really how the cybersecurity industry likes to operates. Maybe Anthropic believes this entire AI thing is a brand new security regime and so believe existing resiliences are moot. That we should all follow blindly as they lead the fight. Their narrative is confusing. Are they being actually transparent or transparency-"coded"?

Re: Anthropic’s paper smells like bullshit

#52

Post by jmkni »

dev_l1x_be wrote: post #35

People grossly underestimate APTs. It is more common than an average IT curious person thinks. I happened to be oncall when one of these guys hacked into Gmail from our infra. It took principal security engineers a few days before they could clearly understand what happened. Multiple zero days, stolen credit cards, massive social campaign to get one of the Google admins click on a funny cat video finally. The investi…

Do you mean APT (Advanced persistent threat)?

Re: Anthropic’s paper smells like bullshit

#53

Post by nalekberov »

I have never taken any AI company seriously, but Anthropic with its attitudes already fed me up to the point that, I deleted my account.

Instead of accusing of China in espionage perhaps they have to think about why they force their users to use phone numbers to register.

Re: Anthropic’s paper smells like bullshit

#54

Post by JKCalhoun »

Says "smells a lot like bullshit" but concludes:

"Look, is it very likely that Threat Actors are using these Agents with bad intentions, no one is disputing that. But this report does not meet the standard of publishing for serious companies."

Title should have been, "I need more info from Anthropic."

Re: Anthropic’s paper smells like bullshit

#55

Post by maddmann »

KaiserPro wrote: post #28

When I worked at a FAANG with a "world leading" AI lab (now run by a teenage data labeller) as an SRE/sysadmin I was asked to use a modified version of a foundation model which was steered towards infosec stuff. We were asked to try and persuade it to help us hack into a mock printer/dodgy linux box. It helped a little, but it wasn't all that helpful. but in terms of coordination, I can't see how it would be useful.…

[flagged]

Re: Anthropic’s paper smells like bullshit

#56

Post by CuriouslyC »

Dumblydorr wrote: post #13

What would AGI actually mean for security? Does it heavily favor attackers or defenders? Even LLM, it may not help much in defense but it could teach attackers a lot right? What if employees gave the LLM info during their use that attackers could then get re-fed and study?

IMO AI favors attackers more than defenders, since it's cost prohibitive for defenders to code scan every version of every piece of software you use routinely for exploits, but not for attackers. Also, social exploits are time consuming, and AI is quite good at automating them, and these can take place outside your security perimeter, so you'll have no way of knowing.

Re: Anthropic’s paper smells like bullshit

#57

Post by oskarkk »

emil-lp wrote: post #3
kkzz99 wrote: post #2

Even Claude thinks the report is bullshit. https://x.com/RnaudBertrand/status/1989636669889560897

Even your own AI model doesn't buy your propaganda Let's not pretend the output of LLMs has any meaningful value when it comes to facts, especially not for recent events.

The LLM was given Anthropic's paper and asked "Is there any evidence or proof whatsoever in the paper that it was indeed conducted by a Chinese state-sponsored group? Answer by yes or no and then elaborate". So the question was not about facts or recent events, but more like a summarizing task, for which an LLM should be good. But the question was specifically about China, while TFA has broader criticism of the paper.
Post reply on HN