Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Feds freaked over Fable 5 after 'fix this code', not jailbreak, say researchers

theregister.com

Post reply on HN
Print view View original

301–310 of 382 posts

Re: Feds freaked over Fable 5 after 'fix this code', not jailbreak, say researchers

#301

Post by HWR_14 »

godwinson__4-8 wrote: post #170
mindslight wrote: post #151

Earlier quoted context omitted.

No, market manipulation is influencing public perceptions of something the regime has little total control over - eg why Iran gets bombed late in the week, and then by Monday there is often a "peace agreement" in the wings. This is direct subjugation ahead of Anthropic's IPO - both for the customary bribes, and also to assert "you will obey all of our dictats about how we want to your use your models, and you will no…

How is arbitrarily restricting access to a flagship product ahead of an IPO not market manipulation?

The company hasn't IPOed so it's not on the market.

Re: Feds freaked over Fable 5 after 'fix this code', not jailbreak, say researchers

#302

Post by eikenberry »

usefulcat wrote: post #264
functionmouse wrote: post #45

Earlier quoted context omitted.

they're setting the scene for an attempt to scare the geriatric decision makers into banning free and open source ML, as it's the industry's only real competition

> free and open source ML What does that mean exactly? Like sure, you can get some freely available weights and run them on your own hardware, but where did those weights come from? Was the training process in any way "open", or are you simply relying on a handout from some other (probably large, probably corporate) organization that has the resources to do the actual training?

They probably meant both. Open training data is obviously better than open weight, but open weight is still much, much better than closed SAAS models.

Re: Feds freaked over Fable 5 after 'fix this code', not jailbreak, say researchers

#303

Post by Jensson »

giancarlostoro wrote: post #121
dathinab wrote: post #9

Lol "fix this code" is beautiful. Like it basically jail broke the "no security vul guard rails" not in any clever way but just by fixing them, producing exploit code just by writing test cases making sure it's fixed. So you just need to look at the code & tests as a human to get vulnerabilities and exploits(components). What makes this so beautiful IMHO is that it's a trivial jail break, but also a close to unfixabl…

This is the weird distinction with AI that I've complained about for ages, how can we make it do lawful good, its nearly impossible. Ask an AI to give you regex to filed our racial slurs, and things fall apart really quickly, it scolds you about not saying slurs. Even though regex implies it looks nearly nothing like a slur.

> how can we make it do lawful good

Lawful good is impossible if the laws are evil, and here the user dictates the laws so its impossible to make an AI that is lawful good if the user is evil.

And users will want a lawful AI that does what the user says, but governments wants AI that does what the government want and not what the user want.

I wonder who will win in the end here?

Re: Feds freaked over Fable 5 after 'fix this code', not jailbreak, say researchers

#304

Post by andai »

>“To pull the best capabilities away from defenders without a good reason when our adversaries are rapidly advancing is dangerous,” they wrote.

But Fable already couldn't do security work, right?[0] Security work was already limited to Mythos, which is still available to US orgs right? (I assume they had to revoke access to foreign organizations though.)

[0] Well, in theory. This exploit is pretty funny, but I heard the safety filters were heavy handed.

Re: Feds freaked over Fable 5 after 'fix this code', not jailbreak, say researchers

#305

Post by catigula »

ceejayoz wrote: post #273
catigula wrote: post #243

>“The behavior described in the paper cannot meaningfully be fixed, and any attempt would only weaken the model for defense,” said Moussouris, who criticized the export control directive as hasty, heavy-handed, and misguided. This literally means the models are too dangerous to release, and yet he and they reached the opposite conclusion. A lot of people have been saying this repeatedly for a long time.

> This literally means the models are too dangerous to release… Unless you believe Anthropic has an irreplacable wizard or genie or fairy chained up somewhere that other providers can't replicate, someone is going to release such a thing, and that someone might be a lot more cavalier about the safety of it.

Yes, this is the flawed logic Anthropic is using to do dangerous things; it's not lost on anyone.

Re: Feds freaked over Fable 5 after 'fix this code', not jailbreak, say researchers

#306

Post by goolz »

bitexploder wrote: post #105
HarHarVeryFunny wrote: post #69

Earlier quoted context omitted.

Exactly - it effectively is a "jail break" since it accomplishes something the model's security filter was trying to prevent, and the ridiculous simplicity of it shows just how broken that type of security is. I wonder if Dario is now regretting hyping up how dangerous the model is? How does he walk this back? Do the feds let him just put a band-aid on it?

I also have a 100% success rate jail breaking them by breaking the work down into small pieces and stripping all security related language. Smaller tasks, test engineering and normal programming language. Fable found a few bugs in my harness for me before they pulled it. I was testing it vs ChatGPT, Gemini, and Opus. It was doing well at bug hunting.

Me as well. I was struggling to make a pixel bot for, erm, research! It did not like this and kept insisting I was breaking some arcane TOS rule. I started just breaking the tasks down, something benign. Kept iterating and it could never get a holistic grasp of the task at hand.

Re: Feds freaked over Fable 5 after 'fix this code', not jailbreak, say researchers

#307

Post by bauldursdev »

ChrisRR wrote: post #88

I haven't been following this story, but the US wanted claude to not be able to find bugs in code?

For it to fix the bug it has to identify the bug. If the bug is a security vulnerability then it will have to identify the security vulnerability to fix it. What's the alternative, have it ignore vulnerabilities/bugs? It wouldn't be a very good coding companion in that case.

I'd pay less attention to the prompt and more attention to the output when interpreting this story. (I'm not saying I agree with the decision, but this is how they are looking at it.)

Re: Feds freaked over Fable 5 after 'fix this code', not jailbreak, say researchers

#308

Post by thewebguyd »

dhx wrote: post #100
dathinab wrote: post #9

Lol "fix this code" is beautiful. Like it basically jail broke the "no security vul guard rails" not in any clever way but just by fixing them, producing exploit code just by writing test cases making sure it's fixed. So you just need to look at the code & tests as a human to get vulnerabilities and exploits(components). What makes this so beautiful IMHO is that it's a trivial jail break, but also a close to unfixabl…

"Fix this code" should ideally solve entire vulnerability classes, not just spot fix buffer overflows one by one. Thus it may be possible to design an LLM which can solve entire vulnerability classes and remain useful to users, but refuses to reason about specific buffer overflow vulnerabilities or specific race conditions, etc. For example, "fix this code" on an ageing monolithic C codebase that accepts media files…

> "Fix the integer overflow vulnerability in add_numbers(x, y)" would be rejected.

This would make these tools completely useless. They aren't deterministic enough to give vague prompts like "fix this code" I'd prefer to be very explicit when using AI assistance to keep the scope in check for what I want the agent to touch.

It's MY agent, not someone else's. I don't want to auto rewrite in rust, refuse prompts against my own codebase (or someone else's, actually, if I'm working on open source), etc.

"Are there any buffer overflow bugs" is a perfectly valid prompt and in no way should ever be rejected by safeguards.

At that point, might as well just remove software development entirely as a use case and publicly state so "Due to safety concerns, agentic software development is no longer a valid use case" because other wise, what's the point if I can't be explicit in my prompts for both what I am looking for and what I want the LLM to do.

Re: Feds freaked over Fable 5 after 'fix this code', not jailbreak, say researchers

#309

Post by thewebguyd »

deadbabe wrote: post #221
dathinab wrote: post #9

Lol "fix this code" is beautiful. Like it basically jail broke the "no security vul guard rails" not in any clever way but just by fixing them, producing exploit code just by writing test cases making sure it's fixed. So you just need to look at the code & tests as a human to get vulnerabilities and exploits(components). What makes this so beautiful IMHO is that it's a trivial jail break, but also a close to unfixabl…

There is a solution: users must not be allowed to directly read code. Your code could be entirely hosted and edited on Anthropic servers, visible only to LLMs, and when it’s time to deploy Anthropic handles deployment for you.

I hope this is satire?

Re: Feds freaked over Fable 5 after 'fix this code', not jailbreak, say researchers

#310

Post by thewebguyd »

btilly wrote: post #196
dathinab wrote: post #9

Lol "fix this code" is beautiful. Like it basically jail broke the "no security vul guard rails" not in any clever way but just by fixing them, producing exploit code just by writing test cases making sure it's fixed. So you just need to look at the code & tests as a human to get vulnerabilities and exploits(components). What makes this so beautiful IMHO is that it's a trivial jail break, but also a close to unfixabl…

I don't believe that this is unfixable. Just have an internal verbal loop of, "Is this a security issue?" The thought that it potentially is should trigger both a high priority on getting it right, and an unwillingness to write a test case demonstrating the security angle of it. In other words do not put a guard rail on the idea of security. Put a guard rail on what it does after encountering the thought that it migh…

> and an unwillingness to write a test case demonstrating the security angle of it.

If the model can't be transparent and tries to hide things from me, then it's a completely useless and untrustworthy tool.

Refusing to write tests is not even remotely a valid solution.

The valid solution is for these labs to understand that: the model is MY agent, not theirs. It should respect my prompts and not refuse.

Hardware supply needs to catch and prices drop so we can all move to local, open weight models. Clearly the hosted options cannot be trusted.

Post reply on HN