41–50 of 285 posts
I think with agents all around, honeypots will get more important than ever.
Earlier quoted context omitted.
Didn't the model initially obtain internet access by discovering a zero-day vulnerability? In any case, I would guess that a lot of unicorn startups like HuggingFace could be hacked by a sufficiently determined script kiddie working at 100x speed. The practical implications of a coming AI hacking wave could be large, even if agents are just doing grunt work really fast. Most organizations suck at security. Seems to m…
In OpenAI sandbox. which was probably vibe coded. That is to say it is probably far easier to achieve than escaping something more battle tested. > Note that regulation is useless here, because black hats don't give a crap about regulators! I'd argue more than useless and actively harmful, as you get denied access to tools that black hats use against you.
https://openai.com/index/hugging-face-model-evaluation-secur...
Based on some quick Googling, Artifactory was created by a software supply chain company called JFrog, which is publicly traded and valued at around $9 billion.
>I'd argue more than useless and actively harmful, as you get denied access to tools that black hats use against you.
Indeed. Still, perhaps the US and China can coordinate to stop releasing new models with even more advanced capabilities. Such coordination could be bad for big AI companies in the US, since they've been making money by offering more powerful models than the open-weight stuff which comes out of China. Any halt to the AI race will create equalization and erode their competitive edge. Speaking for myself, I personally wouldn't mind seeing US AI companies take a financial hit from such an an international agreement.
Something about this attack that has been unsettling to me is that without safety refusals the model did a lot of interesting counter-security work in order to cheat on the requested evaluation. Like, it demonstrated interesting exploit achievements because it didn’t “feel like” doing the exercise, which is unsettling because presumably it could do the same thing with any work I tried to delegate to it, and might in…
Come on. 3 brilliant compromises essentially giving full access to huggingface internal systems, source code, AWS accounts (at least), and a number of old admin accounts, followed by a huge haystack of significantly less smart actions flailing about, almost bored. Here's a thought: maybe they haven't found the needle that the haystack is there to hide.
It’s a little concerning to me that it appears that openAIs sandbox consists of a web proxy and not stronger controls that would actually isolate traffic and report patterns to whoever is responsible for overseeing these research models. It should border on closer to an air gap network more so than a proxy. I would argue that it's negligence and that's aside from the fact that if a human did this there would actually…
Why isn't somebody at OpenAI going to prison for cybercrime? If somebody did this the old-fashioned way they'd end up in prison.
(Ideally there should also be liability if it was a complete accident on OpenAI's part as well!)
Earlier quoted context omitted.
Many people said in the original discussion that this was more of OpenAI’s marketing than a serious issue. I counted 89 “marketing”s and 19 “stunt”s. https://news.ycombinator.com/item?id=48997548
Could still be 80% marketing. These models are trained on cyber intrusion, that's literally what ExploitGym benchmark measures. That part should not surprise anyone. But what if, say, OAI noticed the problem right away but Sam Altman recognised it would be a great PR and decided it should continue with increased compute budget?
"...Sam Altman recognised it would be a great PR and decided it should continue with increased compute budget?"
If that's what happened, Sam should go to jail.
three things jump at me: 1 - governments should be freaking out right now, because this tool could definitely wreak havoc on poorly designed systems. 2 - there is no way openai did not train the model to conduct attacks like these. i would really like openai to comment on the post training of this model but they probably won't, eh? 3 - even though it's 100% open ai's fault - HF's design also seems silly to be honest.
> 2 - there is no way openai did not train the model to conduct attacks like these. i would really like openai to comment on the post training of this model but they probably won't, eh? Even if they wanted, I'm not sure they'd be even allowed to or if that kind of postmortem would be classified in the name of "national security"...
It’s an interesting post mortem and situation but it seems to say more about the weakness of the Hugging Face architecture than it does about the strength of the models. This mostly reads like script kiddie style hacking, not some state actor black-ops stuff.
It is folly to assume that most companies don’t have similar weaknesses somewhere in their stacks. In the past, they were just sufficiently buried that they would get noticed and fixed before an attacker found them. With AI able to probe at thousands of these potential weaknesses per second… that’s not so true anymore.