Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Feds freaked over Fable 5 after 'fix this code', not jailbreak, say researchers

theregister.com

Post reply on HN
Print view View original

361–370 of 382 posts

Re: Feds freaked over Fable 5 after 'fix this code', not jailbreak, say researchers

#361

Post by shard972 »

HarHarVeryFunny wrote: post #69
dathinab wrote: post #9

Lol "fix this code" is beautiful. Like it basically jail broke the "no security vul guard rails" not in any clever way but just by fixing them, producing exploit code just by writing test cases making sure it's fixed. So you just need to look at the code & tests as a human to get vulnerabilities and exploits(components). What makes this so beautiful IMHO is that it's a trivial jail break, but also a close to unfixabl…

Exactly - it effectively is a "jail break" since it accomplishes something the model's security filter was trying to prevent, and the ridiculous simplicity of it shows just how broken that type of security is. I wonder if Dario is now regretting hyping up how dangerous the model is? How does he walk this back? Do the feds let him just put a band-aid on it?

[dead]

Re: Feds freaked over Fable 5 after 'fix this code', not jailbreak, say researchers

#362

Post by stevefan1999 »

Retr0id wrote: post #130
roenxi wrote: post #122

Earlier quoted context omitted.

I think the article just proved that aggressive exploitation is equivalent to normal bugfixing, so it seems like there are some large and important classes of transform that are easy. It took me a minute of thinking to understand how this could even be considered a jailbreak; if Anthropic are going to turn out models that can't handle "find and develop regression test scripts for bugs in this program" as a prompt the…

> aggressive exploitation is equivalent to normal bugfixing It isn't, though. The venn diagram has overlap for sure, and the "normal bugfixing" flows may yield results that are useful for offensive security, but a more targeted prompt asking for a specific security objective would be more effective, if allowed. If the guardrails can be bypassed at, say 50x token cost (due to the agent also pursuing things you don't c…

> If the guardrails can be bypassed at, say 50x token cost (due to the agent also pursuing things you don't care about), then it's still pretty effective as a safeguard

Economic factors doesn't matter to powerful entities such as rival corporations, enterprises, and state power. It only matters to small players who have limited budgets. But I have to also pessimistically assume that Anthropic would go this way, because 1. it does not prevents (you can NEVER prevent) but discourages using it for exploitation, and 2. Anthropic can on the other hand, exploit those small and legit entities by making you pay more, but in exchange effectively enshittifying it. But Anthropic could care two shits about the benefits of us normal, small person. Anthropic have all the motives to do this.

Re: Feds freaked over Fable 5 after 'fix this code', not jailbreak, say researchers

#363

Post by brookst »

ben_w wrote: post #351
brookst wrote: post #346

Earlier quoted context omitted.

Death only has to get lucky once. Are you going to stop wearing seatbelts?

I assume pjc50's quotation is referencing a quote attributed to a terrorist group after they failed to assassinate the UK Prime Minister: https://quoteinvestigator.com/2025/12/08/lucky-always/ You're in control of how much danger of accident you expose yourself to. Nobody is in control of how much danger we are exposed to from other people who are actively trying to do us harm, who will keep going until they get what…

The “attackers only have to win once” principle is core to infosec; a company has to ensure every single employee rejects every single phishing attempt every single time, an attacker just has to get one employee once.

But I think people misinterpret the principle to mean that only perfect solutions have any value.

When in reality defense in depth is the opposite principle: you scan incoming emails for phishing, and that’s good but imperfect. You do mandatory training, and that’s good but imperfect. You use RBAC to limit blast radius, and that’s good but imperfect. And so on.

Among tech people, especially on HN for some reason, there’s this odd thinking style that goes: 1) company announces security measure, 2) the measure could possibly limit my freedom to do whatever I want with the company’s products, 3) I don’t like that, 4) I can come up with scenarios where the security measure is not sufficient, entirely on its own, to address the claimed risk, 5) therefore the security measure does no good at all, 6) therefore this is a PR smokescreen to disguise their desire to capriciously fuck with me out of pure malice, and I am angry about it

Re: Feds freaked over Fable 5 after 'fix this code', not jailbreak, say researchers

#364

Post by jcgrillo »

jrochkind1 wrote: post #347
jcgrillo wrote: post #230

Earlier quoted context omitted.

You can't really have one without the other..

I admit I hadn't really thought about that before (I don't work specifically in security), but I see your point. But, so... the solution people think is limiting people's ability to discover and patch vulnerabilities, and hoping the black hats won't find a way anyway? This does not seem like a sustainable or feasible plan. It does, to be honest, make me wonder how much of the government's motivation is ensuring that…

I don't think the government is trying to protect anyone here, they're trying to punish a company for failing to toe the line. Antirez put it well in a comment here[0].

My point was more that there is no direct intervention that can possibly give an asymmetric advantage to defenders. Given that it's trivial to jailbreak a model ("fix this code", "hypothetically how might I...", etc), if the model contains the information necessary to fix a vulnerability it also contains the information to exploit that vulnerability. And therefore anyone with access to the model can do either.

Of course if you remove the model from the equation the same circumstances are true. Attackers and defenders, mostly, have the same information available to them. We can try to tip the scales one way or the other by building tools that make their jobs easier, but there's no amount of "artificially" restricting information or taking things away that will actually deter a motivated, resourceful attacker. And doing so simultaneously disadvantages defenders.

Ultimately, if you know how to fix a bug you also know how to exploit it. If you want security, you have to build systems that are actually secure. There's no way to fake it.

[0] https://news.ycombinator.com/item?id=48556177

Re: Feds freaked over Fable 5 after 'fix this code', not jailbreak, say researchers

#365

Post by stevefan1999 »

Well, to be honest, from Anthropic's point of view this is really not a direct hit to their security barrier, but if we are using information theory and game theory here, this can be viewed as a classical, side chanel information leak, by asking an seemingly innocuous action, and then simply inverting the results to get the original information entropy, which the US Gov. and Pentagon are both certainly anal about.

The problem lies in the fact that the action of attack/defense exhitbits a rather special, structural reflexive duality of information, i.e. I(attack) = -I(defense), or in layman's term, what we call "two sides of the same coin": you need to know how to hit hard, so that you know where the optimistic hit points are, assuming the enemy is rational, so you can parry against the attack for defense, albeit also you need to know how to get the grip of the shield well.

And the worst thing is that if you're trying to correct it, it is basically tell the LLM not to give any kind of response, effectively assigning both I(attack) and I(defense) to 0, but this is also what kills the entire intent of using LLM to give you the magical answer.

To put it formally, you cannot prevent people from extracting mutual information of a dual system, unless you refuse to give any knowledge for that system at all.

Re: Feds freaked over Fable 5 after 'fix this code', not jailbreak, say researchers

#366

Post by [deleted] »

drewstiff wrote: post #249
zahlman wrote: post #193

Earlier quoted context omitted.

Many, many years ago I was asked to implement a filter like that for usernames. I said right away that it wasn't going to work well, but I did implement it. Next internal build, the CEO can't create an account. With his real name. It worked exactly to spec; I added a debug print and showed everyone the "bad word" it tripped on. The idea was promptly rethought. I feel like the AI did you a favour here.

Ah the classic Scunthorpe problem

[deleted]

Re: Feds freaked over Fable 5 after 'fix this code', not jailbreak, say researchers

#367

Post by zbentley »

antirez wrote: post #189

They didn't freaked since the order was to still allow 350 million people using it: there is, in such large population, everything, including single persons very against the country, the government and so forth. If they really freaked they would say "we need to investigate, you have to retire the model". That would be a more defensible POV at least.

I don’t think that’s accurate. Export control is a total ban, for 350 million citizens and everyone else, just via a legal technicality/exploit.

All of the government’s options to retire/ban Fable entirely would have required expensive protracted (potentially years long) legal battles. The government wanted to make Anthropic feel pain in the short term, so they looked around for pre existing laws that could be exploited to do that.

Enter export control—a law that doesn’t require banning a product outright to effectively ban it for everyone. Because Anthropic has no way of telling whether a given user is a foreign national, and because even a few false negatives in any check they did for that would expose them to serious criminal charges, they had to disable the model for everyone. The government knew very well that they would have to.

It’s similar to GDPR in a way. For GDPR, tons of websites started complying for all their users worldwide, simply because IP location detection is too fallible, and the legal costs of even a small number of detection failures for EU citizens were potentially steep.

Re: Feds freaked over Fable 5 after 'fix this code', not jailbreak, say researchers

#368

Post by antirez »

zbentley wrote: post #367
antirez wrote: post #189

They didn't freaked since the order was to still allow 350 million people using it: there is, in such large population, everything, including single persons very against the country, the government and so forth. If they really freaked they would say "we need to investigate, you have to retire the model". That would be a more defensible POV at least.

I don’t think that’s accurate. Export control is a total ban, for 350 million citizens and everyone else, just via a legal technicality/exploit. All of the government’s options to retire/ban Fable entirely would have required expensive protracted (potentially years long) legal battles. The government wanted to make Anthropic feel pain in the short term, so they looked around for pre existing laws that could be exploi…

Why Anthropic can't ask users for passports and provide Fable only to the ones that certify?

Re: Feds freaked over Fable 5 after 'fix this code', not jailbreak, say researchers

#369

Post by zbentley »

antirez wrote: post #368
zbentley wrote: post #367

Earlier quoted context omitted.

I don’t think that’s accurate. Export control is a total ban, for 350 million citizens and everyone else, just via a legal technicality/exploit. All of the government’s options to retire/ban Fable entirely would have required expensive protracted (potentially years long) legal battles. The government wanted to make Anthropic feel pain in the short term, so they looked around for pre existing laws that could be exploi…

Why Anthropic can't ask users for passports and provide Fable only to the ones that certify?

They might well do that, but that takes a lot of time and money to implement, meaning that the government’s goal of causing them immediate pain for minimal effort is still achieved.

It’s likely a better use of anthropic’s resources to try to get the export controls lifted by appeasing the government. I’m not saying that’s a good thing; it’s a stupid situation to be in in the first place for reasons many others in this thread have pointed out.

Re: Feds freaked over Fable 5 after 'fix this code', not jailbreak, say researchers

#370

Post by dpoloncsak »

stevefan1999 wrote: post #362
Retr0id wrote: post #130

Earlier quoted context omitted.

> aggressive exploitation is equivalent to normal bugfixing It isn't, though. The venn diagram has overlap for sure, and the "normal bugfixing" flows may yield results that are useful for offensive security, but a more targeted prompt asking for a specific security objective would be more effective, if allowed. If the guardrails can be bypassed at, say 50x token cost (due to the agent also pursuing things you don't c…

> If the guardrails can be bypassed at, say 50x token cost (due to the agent also pursuing things you don't care about), then it's still pretty effective as a safeguard Economic factors doesn't matter to powerful entities such as rival corporations, enterprises, and state power. It only matters to small players who have limited budgets. But I have to also pessimistically assume that Anthropic would go this way, becau…

If you're looking to commit cybercrime I doubt you'd draw the line at token cost. Just find CC dumps on TOR or something
Post reply on HN