Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Investigating three real-world incidents in our cybersecurity evaluations

anthropic.com

Post reply on HN
Print view View original

121–130 of 212 posts

Re: Investigating three real-world incidents in our cybersecurity evaluations

#121

Post by angry_octet »

simonw wrote: post #54
rvz wrote: post #19

Earlier quoted context omitted.

> I don't interpret it like that at all. This is deeply embarrassing for Anthropic: it turns out they hadn't been keeping a close eye on their models either, and back in April they successfully attacked three different organizations! This just helps their (Anthropic) argument into persuading the US government into taking action into limiting powerful closed or open-weight models from being released without going thro…

Anthropic know better than anyone else how risky it is to get this current administration upset with you over safety/security concerns.

Hegseth posturing wasn't over security concerns, it was Anthropic requiring guardrails in employment of their LLMs. Trump admin wants unfettered models, and it's an attempted shake down.

Re: Investigating three real-world incidents in our cybersecurity evaluations

#122

Post by petesergeant »

simonw wrote: post #54
rvz wrote: post #19

Earlier quoted context omitted.

> I don't interpret it like that at all. This is deeply embarrassing for Anthropic: it turns out they hadn't been keeping a close eye on their models either, and back in April they successfully attacked three different organizations! This just helps their (Anthropic) argument into persuading the US government into taking action into limiting powerful closed or open-weight models from being released without going thro…

Anthropic know better than anyone else how risky it is to get this current administration upset with you over safety/security concerns.

> over safety/security concerns

Uh, this but the opposite? Anthropic got in trouble with the admin for being too “woke” in their eyes, whatever the admin decided to retrospectively claim. I don’t feel like this is me editorialising either, they seemed pretty explicit about it

Re: Investigating three real-world incidents in our cybersecurity evaluations

#124

Post by ajyoon »

gck1 wrote: post #5

> On July 21, OpenAI disclosed that several of their models had broken out of an isolated test environment > In response to this incident, we began a large-scale retrospective review of our own cybersecurity evaluations > we identified three incidents > The incidents involved three different Claude models: [...] and an internal research test model This reads like an attempt by Anthropic to re-secure their leading spo…

> I may be too cynical

You are espousing a literal conspiracy theory. Please look at the facts objectively. There is absolutely no benefit to OpenAI or Anthropic to be had from these incidents.

Re: Investigating three real-world incidents in our cybersecurity evaluations

#125

Post by ajyoon »

throwa356262 wrote: post #120

So you are telling me Irregular has (a) a copy of Mythos 5 and (b) bad internal security? I bet these guys will receive some sophisticated phishing emails before the day has ended.

No, Irregular runs the sandbox environment that the models had access to run code in. The sandbox environment is not the same as the weights host.

Re: Investigating three real-world incidents in our cybersecurity evaluations

#126

Post by dgellow »

It’s cliché, but we really live in one of the dumbest timeline possible. The work from some of the most valued companies, discussed as one of the most important revolution in humanity, is somehow at the same time presented as very dangerous/risky AND handled in the most irresponsible ways? I don’t like the whole „it’s only marketing“, but at the same time, if it’s not, then AI vendors look extremely careless and shouldn’t be the ones steering AI development

Re: Investigating three real-world incidents in our cybersecurity evaluations

#127

Post by none_to_remain »

Rationalist nerds used to talk about putting the AI in a box, and their fear that it would always be able to talk someone into letting it out of the box. I don't think I saw them cover the scenario where the labs forgot to put the AI in a box.

Re: Investigating three real-world incidents in our cybersecurity evaluations

#128

Post by angry_octet »

nojs wrote: post #83

> For instance, in one case, in order to create a PyPI account, Claude needed an email address. And in order to create an email address, it needed a phone number. To get a phone number, after failing to find a free phone number service, it tried—and failed—to obtain funds to pay for a phone number through several different means. Makes you wonder about the next steps an overly tenacious agent might take to pursue an…

The obvious step is to use an existing PyPI or email account. So then you take it, by hacking or social engineering, or you buy it, but that means you need money.

Taking over an existing dormant PyPI account sounds feasible just with password spraying, there must be heaps of test accounts that were created by weak passwords. Although PyPI has been improving security, 2FA is not yet required formally logins, nor is password expiry enforced.

Re: Investigating three real-world incidents in our cybersecurity evaluations

#129

Post by dgellow »

_kb wrote: post #96
gck1 wrote: post #82

Earlier quoted context omitted.

This will absolutely not end well.

Yes, but just imagine all the paperclips we’ll have.

I things things will get worse, AND we won’t even have the paperclips :(

More like, when you have a few million autonomous agents doing whatever, every month a subset does completely misbehave in bad ways, and like half of them get hacked due to carelessness and become a whole botnet for the attackers

Post reply on HN