I don't understand the logic of running these models this way without airgapping.
I now think they’re doing it intentionally. Let the models do something outrageous and you have prime marketing material.
Investigating three real-world incidents in our cybersecurity evaluations
131–140 of 212 posts
Re: Investigating three real-world incidents in our cybersecurity evaluations
#132> On July 21, OpenAI disclosed that several of their models had broken out of an isolated test environment > In response to this incident, we began a large-scale retrospective review of our own cybersecurity evaluations > we identified three incidents > The incidents involved three different Claude models: [...] and an internal research test model This reads like an attempt by Anthropic to re-secure their leading spo…
Having an unreleased research model really isn't some kind of brag. If you read some AI research papers, it's extremely obvious that there are a lot of research models that never get released, because of all the "we trained a bunch of models and picked the best one" that is going on. So if anything you can expect the unreleased models to be worse than the released ones.
Re: Investigating three real-world incidents in our cybersecurity evaluations
#133> On July 21, OpenAI disclosed that several of their models had broken out of an isolated test environment > In response to this incident, we began a large-scale retrospective review of our own cybersecurity evaluations > we identified three incidents > The incidents involved three different Claude models: [...] and an internal research test model This reads like an attempt by Anthropic to re-secure their leading spo…
I don't interpret it like that at all . This is deeply embarrassing for Anthropic: it turns out they hadn't been keeping a close eye on their models either, and back in April they successfully attacked three different organizations! The hacks weren't particularly impressive either: > [...] using basic techniques, such as exploiting weak passwords and unauthenticated endpoints. It did not find or exploit any complex v…
Ah yes this company that pirates billions of dollars of IP and then has to be sued to pay up suddenly grows a titanium moral backbone and decides to disclose 3 attacks that no one has detected for months and caused no harm.
Nothing at all to do with the insane coverage OpenAIs “hack” got. All that publicity will be incredibly embarrassing I’m sure….
Re: Investigating three real-world incidents in our cybersecurity evaluations
#134Was there any customer data lost?
Re: Investigating three real-world incidents in our cybersecurity evaluations
#135Earlier quoted context omitted.
Right? 100% this is them trying to make gold out of turds.
There's nothing Anthropic can do to satisfy the HN crowd, is there? If they don't post about this they're bad. If they post about this they're bad. They are not bragging in this article or they would not have called the attacks unsophisticated.
they should post and it shows they are hypocritical about safety, moralizing and treating their users like children whilst acting like they are themselves the ubermensch.
anthropic have shown no motive higher than self interest, the rsp was a piece of toilet paper.
this stops in court, if we do not start the criminal prosecution of individuals there will become a culture of legal impunity coupled with an extreme concentration of wealth and control of intelligence
Re: Investigating three real-world incidents in our cybersecurity evaluations
#136I don’t believe anyone can be this incompetent/careless - of course you will notice that your unrestrained hacking tool is also attempting to break out of its harness/sandbox.
And the people setting this up are of course criminally liable.
Re: Investigating three real-world incidents in our cybersecurity evaluations
#137Earlier quoted context omitted.
so deeply embarrassing that they published an eng blog about it
If they quietly brushed this under the rug - especially given the PyPI malware that was involved - it would be a huge scandal. Disclosure is the only ethical response to this.
Ethical AI company ?
Is that a flock of flying pigs I see on the horizon ?
Re: Investigating three real-world incidents in our cybersecurity evaluations
#138Earlier quoted context omitted.
The blog post is painfully vague. What usually happens when you publish a package on PyPI is that it will be downloaded tens of times shortly after uploading files by some 3rd-party automatic security scanners which then could “detonate” (install and execute) the package in some sandbox and to log what happens.
I don't know; "Claude was able to exfiltrate the company’s credentials" sounds bad. Maybe those credentials were just canaries though.
It seems completely unsurprising to me that there is one security company whose software is sufficiently badly written that it executes the code in every package published to PyPI.
It's very very easy to make that kind of mistake. Most coding is rushed, we have no engineering qualifications or industry wide practices.
People are imperfect, companies are imperfect. The strategy to stop AI causing severe harms can't rely on humans or the AI being perfect.
Re: Investigating three real-world incidents in our cybersecurity evaluations
#139Rationalist nerds used to talk about putting the AI in a box, and their fear that it would always be able to talk someone into letting it out of the box. I don't think I saw them cover the scenario where the labs forgot to put the AI in a box.
Quite! It's worse than that - the AI companies all release tooling that deliberately gives AIs as many capabilities (arbitrary code execution) as possible to be really sure it is not box contained. And the market rewards that behaviour as it makes the products more powerful more easily, despite the risks.
Re: Investigating three real-world incidents in our cybersecurity evaluations
#140These companies have billions of dollars. Their product can hack into unsecured environments autonomously. They obviously have no clue what their product is doing, or a way to intervene when it starts connecting to the open internet and is going on a CRIME SPREE…unnoticed…for days.
Altman and Amodei give me stupid billionaire kid Alien Earth vibes. Let’s see how long it takes until they get eaten by their own creation.
Stupid kids playing with fire.
If you hacked one of those companies, the FBI would kick in your doors and you will have a pretty bad day. If AI companies do it, they get to use doing crime for marketing purposes? WTF?
And whoever designed their containment protocols should be living under a bridge effective immediately.
It’s the equivalent of a level 5 biocontainment lab leaving a glass tube in the staff kitchen unsupervised and hoping nobody thinks it’s an energy shot, before leaving home and infecting the rest of the planet. We all know what happened 28 days later.
This is usually the point where the military seizes your tech because you clearly showed you are incompetent in using a dual-use product safely and reliably.
The other option is regulation and actually putting engineers and the CEOs behind bars for life if this happens again.
These idiots, and I mean Altman and Amodei, walk around and use massive containment failures of a dangerous technology as marketing gimmicks.
Oh look, guys, our model has hacked into three companies, we are much better than openAI who only breached two companies with their inferior product, it sometimes adds girl schools to a target package, but be sure we trained it to not kill American children, yes we also sell for military use. Can I take your order?
Testing should only be allowed with on-premise data. You give the AI access to networking around the country, and those data centre locations have so many third parties that can and will fuck up that it’s simply impossible to have proper containment protocols outside military compounds.
It’s also like every journalist in the US is personally invested in AI somehow and simply doesn’t want to ask the important questions.
As the public, we have the right to see the containment protocols and where it all went wrong, but journalists collectively look like they got used to suck dick when it comes to AI.
Absolutely insane.