Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Feds freaked over Fable 5 after 'fix this code', not jailbreak, say researchers

theregister.com

Post reply on HN
Print view View original

211–220 of 382 posts

Re: Feds freaked over Fable 5 after 'fix this code', not jailbreak, say researchers

#211

Post by anuramat »

pjc50 wrote: post #57
martinald wrote: post #6

If you set aside political menace, this is a huge problem with Anthropic's strategy. You _cannot_ say that Mythos is super dangerous and can only be rolled out to certain people, but then release Fable with anything other than bulletproof cyber denials. Clearly with LLMs, bulletproof denials are ~impossible due to the way LLMs work. So you've ended up in a situation where Anthropic are simultaneously claiming it's a…

> Clearly with LLMs, bulletproof denials are ~impossible due to the way LLMs work Exactly. AI safety is nonsensical. You cannot define the set of "bad strings". The billion monkeys with typewriters are eventually going to be able to produce them. Any "safety" system for constraining LLM output is going to have a nonzero leak rate. But on the other hand, this is also irrelevant, unless you're irresponsible enough to c…

is nonzero leak rate sufficient for someone to practically exploit it? if you have to spend $10000 in tokens to get it to do what you want, is it still worth it? what if they manually review the requests of the users that trigger the guardrails too often?

Re: Feds freaked over Fable 5 after 'fix this code', not jailbreak, say researchers

#212

Post by aspenmartin »

btilly wrote: post #196
dathinab wrote: post #9

Lol "fix this code" is beautiful. Like it basically jail broke the "no security vul guard rails" not in any clever way but just by fixing them, producing exploit code just by writing test cases making sure it's fixed. So you just need to look at the code & tests as a human to get vulnerabilities and exploits(components). What makes this so beautiful IMHO is that it's a trivial jail break, but also a close to unfixabl…

I don't believe that this is unfixable. Just have an internal verbal loop of, "Is this a security issue?" The thought that it potentially is should trigger both a high priority on getting it right, and an unwillingness to write a test case demonstrating the security angle of it. In other words do not put a guard rail on the idea of security. Put a guard rail on what it does after encountering the thought that it migh…

Right but the issue is users have full control over context. A security-violating action by a coding agent in one context can be completely innocuous under other contexts etc, or breaking down the task into multiple tasks that in isolation do not violate anything.

Re: Feds freaked over Fable 5 after 'fix this code', not jailbreak, say researchers

#213

Post by shwaj »

an0malous wrote: post #167
HarHarVeryFunny wrote: post #69

Earlier quoted context omitted.

Exactly - it effectively is a "jail break" since it accomplishes something the model's security filter was trying to prevent, and the ridiculous simplicity of it shows just how broken that type of security is. I wonder if Dario is now regretting hyping up how dangerous the model is? How does he walk this back? Do the feds let him just put a band-aid on it?

Cheapest option is to gift an enormous golden statue of Trump for his ballroom

“Put it there in the back with the others”, lol.

Re: Feds freaked over Fable 5 after 'fix this code', not jailbreak, say researchers

#215

Post by lachlan_gray »

btilly wrote: post #196
dathinab wrote: post #9

Lol "fix this code" is beautiful. Like it basically jail broke the "no security vul guard rails" not in any clever way but just by fixing them, producing exploit code just by writing test cases making sure it's fixed. So you just need to look at the code & tests as a human to get vulnerabilities and exploits(components). What makes this so beautiful IMHO is that it's a trivial jail break, but also a close to unfixabl…

I don't believe that this is unfixable. Just have an internal verbal loop of, "Is this a security issue?" The thought that it potentially is should trigger both a high priority on getting it right, and an unwillingness to write a test case demonstrating the security angle of it. In other words do not put a guard rail on the idea of security. Put a guard rail on what it does after encountering the thought that it migh…

I think they were doing something like this, the tradeoff is that it's hard to do without an irritating number of false positives and/or wasting loads of precious tokens on useless audits.

Re: Feds freaked over Fable 5 after 'fix this code', not jailbreak, say researchers

#216

Post by btilly »

torben-friis wrote: post #210
btilly wrote: post #196

Earlier quoted context omitted.

I don't believe that this is unfixable. Just have an internal verbal loop of, "Is this a security issue?" The thought that it potentially is should trigger both a high priority on getting it right, and an unwillingness to write a test case demonstrating the security angle of it. In other words do not put a guard rail on the idea of security. Put a guard rail on what it does after encountering the thought that it migh…

The end result of that is that your model can't fix or acknowledge security issues for fear of disclosing them. This is the beauty the above poster mentioned: the ability to improve code is inherently coupled with the ability to recognize its shortcomings. You can't have one without the other.

What I suggested would allow it to fix the issues. Just not write a test that was directly usable as a security exploit.

This doesn't stop attackers from being able to leverage the analysis. But it does make the tool more useful for defenders than attackers. Which is the best that you can hope for from a useful tool.

Re: Feds freaked over Fable 5 after 'fix this code', not jailbreak, say researchers

#218

Post by anuramat »

scotty79 wrote: post #117
embedding-shape wrote: post #19

> “‘Fix this code,’ plus several manual steps to generate test scripts, Feels like the title isn't really giving the full context of what they ended up actually seeing, despite what the lede implies multiple times. Still, ban seems stupid... Still no actual leak of the full "third-party research paper"?

If what your patch fixes is a vulnerability bug then the test for it is basically an exploit.

isn't there a pretty big gap between a segfault and an rce? I thought that was the entire point -- that mythos closed the gap

Re: Feds freaked over Fable 5 after 'fix this code', not jailbreak, say researchers

#219

Post by btilly »

Kinrany wrote: post #202
btilly wrote: post #196

Earlier quoted context omitted.

I don't believe that this is unfixable. Just have an internal verbal loop of, "Is this a security issue?" The thought that it potentially is should trigger both a high priority on getting it right, and an unwillingness to write a test case demonstrating the security angle of it. In other words do not put a guard rail on the idea of security. Put a guard rail on what it does after encountering the thought that it migh…

That would make the model useless

How does this make the model useless? It finds and fixes the security hole. It can even write a test that verifies that the fix didn't break things. But it deliberately doesn't reveal the fact that it was a security issue that was fixed.

Seems useful to me. But more useful for defenders than attackers.

Re: Feds freaked over Fable 5 after 'fix this code', not jailbreak, say researchers

#220

Post by WarOnPrivacy »

giancarlostoro wrote: post #208
zahlman wrote: post #193

Earlier quoted context omitted.

Many, many years ago I was asked to implement a filter like that for usernames. I said right away that it wasn't going to work well, but I did implement it. Next internal build, the CEO can't create an account. With his real name. It worked exactly to spec; I added a debug print and showed everyone the "bad word" it tripped on. The idea was promptly rethought. I feel like the AI did you a favour here.

Now I'm trying to figure out which word that would be, but yeah. That reminds me of a bug I fixed where my bosses boss found it, we did everything, my boss at the time forced us to deploy anything and call it fixed. Then someone else saw it half a year later, I finally figured out the root cause and fixed it (localStorage vs sessionStorage) and my boss was acting like he didn't know what I was talking about, but I co…

> Now I'm trying to figure out which word that would be

I once had Shi Tao as part of an email username. It tripped filters periodically.

Post reply on HN