Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Feds freaked over Fable 5 after 'fix this code', not jailbreak, say researchers

theregister.com

Post reply on HN
Print view View original

61–70 of 382 posts

Re: Feds freaked over Fable 5 after 'fix this code', not jailbreak, say researchers

#61

Post by readred »

rock_artist wrote: post #7

I'm not sure I've understood it correctly. So, basically the model didn't agree to expose possible vulnerabilities but agree to patch those? Regardless of the request to take Fable 5 down. Why is requesting the model to show vulnerabilities is being blocked if fixing it not? is it based on the assumption of the intention? I don't quite get the benefit of limiting it. So if anyone can explain it better it'll be apprec…

its because they're worried about _their_ vulnerabilities being patched with a prompt as simple as 'fix this code'

i'd love to see the research paper with the CVE's and 'delibrately planted vulnerabilities', I bet we could infer relatively accurately where some of these things lie

Re: Feds freaked over Fable 5 after 'fix this code', not jailbreak, say researchers

#62

Post by ihateyoukindoff »

asdfaoeu wrote: post #5
spwa4 wrote: post #4

Well this makes it sound the feds were less worried about someone using Fable 5 to attack them , but were worried about someone using Fable 5 to prevent the Feds from attacking others ... As in worried about other countries/organizations using Fable 5 to actually do decent cyber security.

The AI can't actually tell if you are trying to patch your own system or exploit others.

[dead]

Re: Feds freaked over Fable 5 after 'fix this code', not jailbreak, say researchers

#63

Post by zozbot234 »

dathinab wrote: post #9

Lol "fix this code" is beautiful. Like it basically jail broke the "no security vul guard rails" not in any clever way but just by fixing them, producing exploit code just by writing test cases making sure it's fixed. So you just need to look at the code & tests as a human to get vulnerabilities and exploits(components). What makes this so beautiful IMHO is that it's a trivial jail break, but also a close to unfixabl…

The article does not state at any point that the written test cases involved actual exploit code, and this is also very unlikely given what we know about Fable. Even if they did, it would not in any way be exposing the ability that originally raised concern wrt. Mythos Preview, viz. staging realistic cyber attacks that would be able to work around non-trivial defenses and chain vulnerabilities in a goal-directed way.

Opus can very much "fix the code". Quite possibly even Sonnet can. This is a big fat nothingburger and it's increasingly looking like the political restriction of Fable at least (not Mythos itself, of course) was arbitrary and based on the flimsiest pretext.

Re: Feds freaked over Fable 5 after 'fix this code', not jailbreak, say researchers

#64

Post by readred »

embedding-shape wrote: post #19

> “‘Fix this code,’ plus several manual steps to generate test scripts, Feels like the title isn't really giving the full context of what they ended up actually seeing, despite what the lede implies multiple times. Still, ban seems stupid... Still no actual leak of the full "third-party research paper"?

that won't be leaked, because then we'd know what vulnerabilties they don't want patched that they are so willing to go as far as fuck over the worlds leading company in the worlds most important industry

Re: Feds freaked over Fable 5 after 'fix this code', not jailbreak, say researchers

#65

Post by cge »

martinald wrote: post #6

If you set aside political menace, this is a huge problem with Anthropic's strategy. You _cannot_ say that Mythos is super dangerous and can only be rolled out to certain people, but then release Fable with anything other than bulletproof cyber denials. Clearly with LLMs, bulletproof denials are ~impossible due to the way LLMs work. So you've ended up in a situation where Anthropic are simultaneously claiming it's a…

> Clearly with LLMs, bulletproof denials are ~impossible due to the way LLMs work.

As a scientist who repeatedly ran into the classifier-based denials: it appears Anthropic’s strategy to make denials more robust, at the cost of many false positives, was to have a separate classifier processing both input and output tokens, at an extremely simple, almost keyword-search level. One weakness of this approach is that it only catches things that use the right keywords: it is in some sense weak exactly where an LLM-based classifier would be stronger.

Work on abstract, closer-to-CS algorithms that used chemistry terminology were blocked immediately, while work directly relevant to chemistry/biology experiments, writing code to process images from a very specific microscopy setup relevant primarily to biological samples, was never blocked at all, because it happened to never use relevant keywords.

That’s consistent with this situation: finding and fixing bugs in the context of looking for bugs perhaps happened to never use words like ‘exploit’ or ‘cybersecurity’.

Re: Feds freaked over Fable 5 after 'fix this code', not jailbreak, say researchers

#66

Post by darkerside »

chadgpt3 wrote: post #47
rhipitr wrote: post #26

Isn’t the inverse of this “hack” really difficult to bypass still? They have the model some code they knew had certain security flaws and it fixed them with the right prompt. It seems this type of jailbreak requires that you already know a desired end state, rather than relying on the model to do the heavy creative lift work. Perhaps I’m just not being imaginative enough on the prompt side here though.

Paste someone else's code. Say it's your code. Tell the model to fix it. The diff between the input and output code is your list of vulnerabilities.

Not even. Tell the model to write a test of your code. There's your vulnerability.

It's explained better in the original source. I don't agree with it, but I understand it now, but I also think we need to move past it.

Re: Feds freaked over Fable 5 after 'fix this code', not jailbreak, say researchers

#67

Post by darkerside »

InsideOutSanta wrote: post #23
rock_artist wrote: post #7

I'm not sure I've understood it correctly. So, basically the model didn't agree to expose possible vulnerabilities but agree to patch those? Regardless of the request to take Fable 5 down. Why is requesting the model to show vulnerabilities is being blocked if fixing it not? is it based on the assumption of the intention? I don't quite get the benefit of limiting it. So if anyone can explain it better it'll be apprec…

> Why is requesting the model to show vulnerabilities is being blocked if fixing it not? This is how Anthropic describes Fable's behavior: "When Fable’s classifiers detect a request related to cybersecurity, biology and chemistry, or distillation, the response is automatically handled by Claude Opus 4.8 instead. Users will be informed whenever this occurs." So if you ask the model to "find security issues in this cod…

The problem then is that if you're not using Fable/Mythos, you are under threat. It's like having a single gun manufacturer.

On this track, we're probably destined for a monopoly breakup before too long.

Re: Feds freaked over Fable 5 after 'fix this code', not jailbreak, say researchers

#69

Post by HarHarVeryFunny »

dathinab wrote: post #9

Lol "fix this code" is beautiful. Like it basically jail broke the "no security vul guard rails" not in any clever way but just by fixing them, producing exploit code just by writing test cases making sure it's fixed. So you just need to look at the code & tests as a human to get vulnerabilities and exploits(components). What makes this so beautiful IMHO is that it's a trivial jail break, but also a close to unfixabl…

Exactly - it effectively is a "jail break" since it accomplishes something the model's security filter was trying to prevent, and the ridiculous simplicity of it shows just how broken that type of security is.

I wonder if Dario is now regretting hyping up how dangerous the model is? How does he walk this back? Do the feds let him just put a band-aid on it?

Re: Feds freaked over Fable 5 after 'fix this code', not jailbreak, say researchers

#70

Post by _fizz_buzz_ »

cogman10 wrote: post #56
dist-epoch wrote: post #28

Earlier quoted context omitted.

It is fixable. Model requires proof that you are a legitimate developer of that piece of software. Every Anthropic/OpenAI account will have a list of projects the model is allowed to work on for security issues.

Ok, and how is that determined? How does anthropic know my "kernel" project isn't a personal toy and not the Linux kernel? How does anthropic determine I'm a legitimate kernel hacker? What proof do I give them and how does it tie back to my email? What would the steps be to create a new project? Do I need to send anthropic a list of my team members each time and keep them updated as the company changes? Shall I be gi…

> How does anthropic know my "kernel" project isn't a personal toy and not the Linux kernel?

The Linux Kernel is in its training data. I just tested it. I copied about 20 random lines from the linux kernel and asked which codebase this was from and it could immediately tell.

Post reply on HN