Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Feds freaked over Fable 5 after 'fix this code', not jailbreak, say researchers

theregister.com

Post reply on HN
Print view View original

161–170 of 382 posts

Re: Feds freaked over Fable 5 after 'fix this code', not jailbreak, say researchers

#161

Post by kordlessagain »

bitexploder wrote: post #105
HarHarVeryFunny wrote: post #69

Earlier quoted context omitted.

Exactly - it effectively is a "jail break" since it accomplishes something the model's security filter was trying to prevent, and the ridiculous simplicity of it shows just how broken that type of security is. I wonder if Dario is now regretting hyping up how dangerous the model is? How does he walk this back? Do the feds let him just put a band-aid on it?

I also have a 100% success rate jail breaking them by breaking the work down into small pieces and stripping all security related language. Smaller tasks, test engineering and normal programming language. Fable found a few bugs in my harness for me before they pulled it. I was testing it vs ChatGPT, Gemini, and Opus. It was doing well at bug hunting.

I took an assembler class in college. Before that, I'd been messing around with Core Wars and working my way through Peter Norton's book on assembly. So when an assignment came up, I used self modifying code to solve it. It was the shortest solution, it ran perfectly, and I submitted it.

The next day, the professor caught me in the math department office (my dad worked there) and said she wanted to talk. Once we were in her office, she told me I wasn't allowed to use self modifying code. I pushed back: "Nothing in the assignment said I couldn't, and the output is correct."

The next class, she walked in and announced that self modifying code was no longer allowed on any assignment. Then she handed back the graded work and I'd gotten a 100.

Thinking back on that: about a week and a half ago I asked Antigravity to build a modern GPU version of Core Wars, except with Redcode mapped directly onto the GPU instruction set. I've had some good success and it's more or less working now, though visualizing what's happening at the GPU/Redcode level is much harder.

But before Fable 5 got yanked, I asked it to "fix" the project and it refused, flipping straight to Opus 4.8. Every single request I sent triggered the fallback. I spent over an hour trying different angles, and I even turned Antigravity loose on automatic so it was the one talking to Fable 5 same result. Every exchange tripped the fallback to 4.8. I wish I'd recorded it.

I also tried a variety of direct requests in a fresh directory "build simple self modifying assembler code" or just "self modifying assembler" and it would switch to 4.8 immediately. It was almost laughable.

There's ZERO credibility to any of these stories right now. If Anthropic really sent something over to this security person, and it's what she says it is, then why on earth didn't they just blog about it?

Hubris is a thing. Companies would do well to remember Steve Jobs in the early Apple days: ship early, ship often, and above all take responsibility for what you ship even when it's broken. Code, hardware, the whole kit all of it can be fixed. Trust is much harder to repair. Anthropic has lost mine, and while I may use them from time to time, it'll be in limited ways.

Re: Feds freaked over Fable 5 after 'fix this code', not jailbreak, say researchers

#162

Post by superice »

chadgpt3 wrote: post #47
rhipitr wrote: post #26

Isn’t the inverse of this “hack” really difficult to bypass still? They have the model some code they knew had certain security flaws and it fixed them with the right prompt. It seems this type of jailbreak requires that you already know a desired end state, rather than relying on the model to do the heavy creative lift work. Perhaps I’m just not being imaginative enough on the prompt side here though.

Paste someone else's code. Say it's your code. Tell the model to fix it. The diff between the input and output code is your list of vulnerabilities.

But this is already how open source works today. If you have the code, you, a human, could find and 'fix' or exploit vulnerabilities as much as you want.

Now if Fable had an easy jailbreak like this that allowed you to attack remote targets that'd be a different story but I genuinely cannot see how neutering its abilities to 'fix' code you already have access to is sensible. It would destroy the value of the model. And don't forget, any actor not abiding by the same rules could develop an model for offensive use just fine, so this protects you against exactly nothing but does destroy a potential defense.

In the end this all comes down to legislation, in much the same way platforms are not responsible for copyright violations IF they abide by some rules, the same has to happen for AI providers. If you have a process for reporting 'jailbreaks' on illegal actions, and prevent users doing illegal stuff on a best effort basis, the rest of it should really just be individual responsibility. If a user wants to use an LLM to crack systems, fine, that's already illegal.

If Tesla FSD deliberately hit somebody, holding Tesla liable is fine. If you messed with FSD until you finally got it to hit a person, then you should be liable. Outlawing FSD because it could theoretically be tampered with is just an odd stance imho.

Re: Feds freaked over Fable 5 after 'fix this code', not jailbreak, say researchers

#163

Post by smallerize »

AndrewKemendo wrote: post #142

I’m still not buying that this was an actual USG order. The only people commenting are “experts” and there has been no official announcement from the USG. This doesn’t smell like a NSL and there’s no process to selectively “export control” something like this. Even so there’s a dozen mechanisms through courts to challenge this, and Anthropic isn’t taking any of them. I think this is a made up crisis for PR with no ac…

David Sacks is on the record confirming it. https://www.tomshardware.com/tech-industry/artificial-intell...

Re: Feds freaked over Fable 5 after 'fix this code', not jailbreak, say researchers

#164

Post by applfanboysbgon »

ceejayoz wrote: post #34
dist-epoch wrote: post #32

Earlier quoted context omitted.

sure. how many cases like these we had so far? 1, 2? and how long did they work to get commit access?

> how many cases like these we had so far? As with clever, careful serial killers, it's tough to count the ones we haven't caught.

It's not that tough. You can get an idea by how many people are being murdered. A successful serial killer results in dead people, and a successful infiltration results in malware being executed. If there are no murdered people with unattributed causes of death, or there are no open-source projects with unattributed causes of malware being shipped, you can conclude there are roughly 0 active serial killers / infiltrators.

It's possible there are infiltrators who are still working on long-term infiltration and haven't yet attempted to add any malicious code anywhere, but the point is that in terms of actual attempts, we've seen a single one and it wasn't even successful despite years of prep.

Re: Feds freaked over Fable 5 after 'fix this code', not jailbreak, say researchers

#165

Post by AmblingAvocado »

asdfaoeu wrote: post #5
spwa4 wrote: post #4

Well this makes it sound the feds were less worried about someone using Fable 5 to attack them , but were worried about someone using Fable 5 to prevent the Feds from attacking others ... As in worried about other countries/organizations using Fable 5 to actually do decent cyber security.

The AI can't actually tell if you are trying to patch your own system or exploit others.

It seems like ... it's not illegal to find exploits, it's illegal to use them. Enforcement should start there, not the nanny state approach that you might do something bad with information. It breaks down a little bit because it means there will be a period of disruption while the bad guys use exploits - but that's already illegal, and the good guys have had time to use the tool & fix things before it went public, right?

Re: Feds freaked over Fable 5 after 'fix this code', not jailbreak, say researchers

#167

Post by an0malous »

HarHarVeryFunny wrote: post #69
dathinab wrote: post #9

Lol "fix this code" is beautiful. Like it basically jail broke the "no security vul guard rails" not in any clever way but just by fixing them, producing exploit code just by writing test cases making sure it's fixed. So you just need to look at the code & tests as a human to get vulnerabilities and exploits(components). What makes this so beautiful IMHO is that it's a trivial jail break, but also a close to unfixabl…

Exactly - it effectively is a "jail break" since it accomplishes something the model's security filter was trying to prevent, and the ridiculous simplicity of it shows just how broken that type of security is. I wonder if Dario is now regretting hyping up how dangerous the model is? How does he walk this back? Do the feds let him just put a band-aid on it?

Cheapest option is to gift an enormous golden statue of Trump for his ballroom

Re: Feds freaked over Fable 5 after 'fix this code', not jailbreak, say researchers

#168

Post by ceejayoz »

applfanboysbgon wrote: post #164
ceejayoz wrote: post #34

Earlier quoted context omitted.

> how many cases like these we had so far? As with clever, careful serial killers, it's tough to count the ones we haven't caught.

It's not that tough. You can get an idea by how many people are being murdered. A successful serial killer results in dead people, and a successful infiltration results in malware being executed. If there are no murdered people with unattributed causes of death, or there are no open-source projects with unattributed causes of malware being shipped, you can conclude there are roughly 0 active serial killers / infiltra…

> You can get an idea by how many people are being murdered.

No, we can't, as that happens a lot via non-serial killers.

A truly successful serial killer is likely one who hides in that noise. No taunting the cops, distributed geographic locations, random methods, avoiding calling cards, and careful not to leave too many traces.

It seems likely that some of the 350k unsolved homicides in the US can be explained this way.

> It's possible there are infiltrators who are still working on long-term infiltration and haven't yet attempted to add any malicious code anywhere…

Or the code's already there, latent, as it would've been in the XZ case, which got discovered by chance and someone very dedicated to looking into a performance glitch.

Re: Feds freaked over Fable 5 after 'fix this code', not jailbreak, say researchers

#170

Post by godwinson__4-8 »

mindslight wrote: post #151
godwinson__4-8 wrote: post #92

Earlier quoted context omitted.

Two words: market manipulation

No, market manipulation is influencing public perceptions of something the regime has little total control over - eg why Iran gets bombed late in the week, and then by Monday there is often a "peace agreement" in the wings. This is direct subjugation ahead of Anthropic's IPO - both for the customary bribes, and also to assert "you will obey all of our dictats about how we want to your use your models, and you will no…

How is arbitrarily restricting access to a flagship product ahead of an IPO not market manipulation?
Post reply on HN