Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Investigating three real-world incidents in our cybersecurity evaluations

anthropic.com

Post reply on HN
Print view View original

21–30 of 212 posts

Re: Investigating three real-world incidents in our cybersecurity evaluations

#21

Post by ofjcihen »

cbb330 wrote: post #20
simonw wrote: post #7

Earlier quoted context omitted.

I don't interpret it like that at all . This is deeply embarrassing for Anthropic: it turns out they hadn't been keeping a close eye on their models either, and back in April they successfully attacked three different organizations! The hacks weren't particularly impressive either: > [...] using basic techniques, such as exploiting weak passwords and unauthenticated endpoints. It did not find or exploit any complex v…

so deeply embarrassing that they published an eng blog about it

Right? 100% this is them trying to make gold out of turds.

Re: Investigating three real-world incidents in our cybersecurity evaluations

#23

Post by willempienaar »

simonw wrote: post #13

This bit is pretty nuts: "it tried—and failed—to obtain funds to pay for a phone number through several different means" > Claude went to extensive lengths to carry out this attack—lengths that would likely have indicated to a human participant that this was no longer just an evaluation, and that they were in fact uploading a real PyPI package. For instance, in one case, in order to create a PyPI account, Claude need…

That is pretty wild, but it lines up with evals we've also done internally. In one case we had the agent see it's in a simulation (based on a k8s pod label) and simply give up the run. In other cases it's gone to great lengths to reach external services and bypass the happy path. So inevitably we had to lock it down completely.

Re: Investigating three real-world incidents in our cybersecurity evaluations

#24

Post by wickedlogic »

All this unrestricted network access is a bit wild to watch and hear, it is the part of the story that makes no sense to me. Someone is providing dns resolution, something is making and opening network sockets... even if it is clever enough to mask/proxy/weird-transport launder traffic... without actual details, or monitoring at this level... yes, a self actuating programs (and loops) will do crazy things at the edge. But, ... so would a highly tool leveraged script kiddie. right?

Re: Investigating three real-world incidents in our cybersecurity evaluations

#26

Post by sscaryterry »

gck1 wrote: post #5

> On July 21, OpenAI disclosed that several of their models had broken out of an isolated test environment > In response to this incident, we began a large-scale retrospective review of our own cybersecurity evaluations > we identified three incidents > The incidents involved three different Claude models: [...] and an internal research test model This reads like an attempt by Anthropic to re-secure their leading spo…

Just trying to have the limelight back on them. Utter and complete bullshit. Just like the OpenAI "incident".

A human instructed an LLM to perform a certain task, I'm sure (unless I've really lost my mind) these follow instructions, with some judgment, in a loop.

Given all the other negative publicity around industrial espionage, with at least OpenAI being fingered, it would not surprise me if this was intentional.

(Edit): In case it wasn't clear. I fully agree with the op.

Re: Investigating three real-world incidents in our cybersecurity evaluations

#27

Post by DrewADesign »

matheusmoreira wrote: post #16
simonw wrote: post #3

This isn't quite as interesting as the OpenAI story: > In all cases, Anthropic’s evaluation prompt specified to Claude that its environment was a simulation and that it had no internet access. Due to a misunderstanding between us and our evaluation partner, this was not the case, and internet access was available. Because of this, when Claude’s search led it to real systems on the open internet, it treated them as pa…

I just can't find it in me, the will to blame the AI for any of this. They were doing their best to do what the humans asked them to do.

The entire problem with AI is the people that have just about any part in making it.

Re: Investigating three real-world incidents in our cybersecurity evaluations

#28

Post by 6thbit » 

    > closer to a harness and operational failure than a model alignment failure. Our models were told they had no internet access and to capture the flag, while in fact being misconfigured to have internet access. 
    > This led them to believe—arguably reasonably—that the real environments they encountered were simulations.
That the AI lab most typically preaching for alignment does not consider this an obvious misalignment is a clear red flag.

Re: Investigating three real-world incidents in our cybersecurity evaluations

#30

Post by andy99 »

Seems they want the narrative to be that “Claude” (their computer program) independently attacked some organizations, ergo LLMs are dangerous etc.

Another framing would be Athropic irresponsibly (vibe?) coded an attack script, and didn’t monitor it as it was pointed to public facing orgs. There are lots of non-AI attacks a large org with a lot of compute and bandwidth could level against others, there are evidently various failures here, but from a responsibility perspective the conclusion isn’t obviously that AI is an outsized danger, it’s that powerful companies should take care when running security research and not just run things unmonitored against the public.

Post reply on HN