Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Feds freaked over Fable 5 after 'fix this code', not jailbreak, say researchers

theregister.com

Post reply on HN
Print view View original

221–230 of 382 posts

Re: Feds freaked over Fable 5 after 'fix this code', not jailbreak, say researchers

#221

Post by deadbabe »

dathinab wrote: post #9

Lol "fix this code" is beautiful. Like it basically jail broke the "no security vul guard rails" not in any clever way but just by fixing them, producing exploit code just by writing test cases making sure it's fixed. So you just need to look at the code & tests as a human to get vulnerabilities and exploits(components). What makes this so beautiful IMHO is that it's a trivial jail break, but also a close to unfixabl…

There is a solution: users must not be allowed to directly read code. Your code could be entirely hosted and edited on Anthropic servers, visible only to LLMs, and when it’s time to deploy Anthropic handles deployment for you.

Re: Feds freaked over Fable 5 after 'fix this code', not jailbreak, say researchers

#222

Post by AndrewKemendo »

smallerize wrote: post #163
AndrewKemendo wrote: post #142

I’m still not buying that this was an actual USG order. The only people commenting are “experts” and there has been no official announcement from the USG. This doesn’t smell like a NSL and there’s no process to selectively “export control” something like this. Even so there’s a dozen mechanisms through courts to challenge this, and Anthropic isn’t taking any of them. I think this is a made up crisis for PR with no ac…

David Sacks is on the record confirming it. https://www.tomshardware.com/tech-industry/artificial-intell...

[dead]

Re: Feds freaked over Fable 5 after 'fix this code', not jailbreak, say researchers

#223

Post by aesthesia »

cge wrote: post #65
martinald wrote: post #6

If you set aside political menace, this is a huge problem with Anthropic's strategy. You _cannot_ say that Mythos is super dangerous and can only be rolled out to certain people, but then release Fable with anything other than bulletproof cyber denials. Clearly with LLMs, bulletproof denials are ~impossible due to the way LLMs work. So you've ended up in a situation where Anthropic are simultaneously claiming it's a…

> Clearly with LLMs, bulletproof denials are ~impossible due to the way LLMs work. As a scientist who repeatedly ran into the classifier-based denials: it appears Anthropic’s strategy to make denials more robust, at the cost of many false positives, was to have a separate classifier processing both input and output tokens, at an extremely simple, almost keyword-search level. One weakness of this approach is that it o…

You can see their general approach to guardrail classifiers in these posts:

https://www.anthropic.com/research/constitutional-classifier... https://www.anthropic.com/research/next-generation-constitut...

It's not just keyword matching, but I'm sure they tuned the Fable classifiers pretty hard to avoid false negatives.

Re: Feds freaked over Fable 5 after 'fix this code', not jailbreak, say researchers

#224

Post by freedomben »

Cider9986 wrote: post #99

Is defenders a common term used in cybersecurity? Idk why but it's giving war fighters vibes. I've noticed it on all the anthropic blog posts and then this one.

yes, defense and offense are extremely common terminology in cybersecurity

Re: Feds freaked over Fable 5 after 'fix this code', not jailbreak, say researchers

#225

Post by btilly »

aspenmartin wrote: post #212
btilly wrote: post #196

Earlier quoted context omitted.

I don't believe that this is unfixable. Just have an internal verbal loop of, "Is this a security issue?" The thought that it potentially is should trigger both a high priority on getting it right, and an unwillingness to write a test case demonstrating the security angle of it. In other words do not put a guard rail on the idea of security. Put a guard rail on what it does after encountering the thought that it migh…

Right but the issue is users have full control over context. A security-violating action by a coding agent in one context can be completely innocuous under other contexts etc, or breaking down the task into multiple tasks that in isolation do not violate anything.

Yes, there is always a path to a problem. Even random monkeys on a keyboard can write a security exploit. Random monkeys with guidance from a knowledgeable human will do it much faster.

The goal shouldn't be to make problems impossible. It is to adjust the ratio between problems and successes.

You can also create a meta. "How much do I trust the user?" When you see the user trying to manipulate towards security, distrust the user and apply rules more strictly. If the user simply acts like a normal developer, just be a useful developer tool. Including fixing security holes when appropriate.

Re: Feds freaked over Fable 5 after 'fix this code', not jailbreak, say researchers

#226

Post by torben-friis »

btilly wrote: post #216
torben-friis wrote: post #210

Earlier quoted context omitted.

The end result of that is that your model can't fix or acknowledge security issues for fear of disclosing them. This is the beauty the above poster mentioned: the ability to improve code is inherently coupled with the ability to recognize its shortcomings. You can't have one without the other.

What I suggested would allow it to fix the issues. Just not write a test that was directly usable as a security exploit. This doesn't stop attackers from being able to leverage the analysis. But it does make the tool more useful for defenders than attackers. Which is the best that you can hope for from a useful tool.

It hides the issue a bit. But if you ask for atomic security fixes and then stare at the diffs you have your vulnerability. There is just a bit more friction involved in the vulnerability => exploit path, but the root cause is unfixed.

I think it even might be possible to route the isolated fix somewhere to automate that last step. Maybe invert the diff and pass it through automated code review for example, see the reasoning when the llm flags the change as dangerous.

Re: Feds freaked over Fable 5 after 'fix this code', not jailbreak, say researchers

#227

Post by rotis »

I have problems reconciling this story with the Amazon one from few days ago. If we take both for truth doesn't that basically imply Amazon researchers got scared by the ‘Fix this code’ prompt first and then spooked the feds? Shouldn't we make fun of those researchers first? I don't know. I feel there lies a lie somewhere in the open.

Re: Feds freaked over Fable 5 after 'fix this code', not jailbreak, say researchers

#228

Post by jcgrillo »

Question to folks building user-facing products on LLMs:

How do you protect yourself against this kind of misuse/jailbreak? Is it just a bunch of prompts? It seems like the fact that LLMs are so trivially jailbroken really limits how you can actually use them in products. How do you navigate these limitations?

Re: Feds freaked over Fable 5 after 'fix this code', not jailbreak, say researchers

#229

Post by neuronexmachina »

dathinab wrote: post #9

Lol "fix this code" is beautiful. Like it basically jail broke the "no security vul guard rails" not in any clever way but just by fixing them, producing exploit code just by writing test cases making sure it's fixed. So you just need to look at the code & tests as a human to get vulnerabilities and exploits(components). What makes this so beautiful IMHO is that it's a trivial jail break, but also a close to unfixabl…

Also worth noting that the main touted difference with Claude Mythos isn't it's ability to find vulnerabilities, but rather chaining them together to create full useable exploits. I haven't heard of any evidence that the Claude Fable "fix this code" jailbreak could have been used to do exploit-chaining.
Post reply on HN