Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Feds freaked over Fable 5 after 'fix this code', not jailbreak, say researchers

theregister.com

Post reply on HN
Print view View original

41–50 of 382 posts

Re: Feds freaked over Fable 5 after 'fix this code', not jailbreak, say researchers

#41

Post by brookst »

ceejayoz wrote: post #31
dist-epoch wrote: post #28

Earlier quoted context omitted.

It is fixable. Model requires proof that you are a legitimate developer of that piece of software. Every Anthropic/OpenAI account will have a list of projects the model is allowed to work on for security issues.

https://en.wikipedia.org/wiki/XZ_Utils_backdoor > A subsequent investigation found that the campaign to insert the backdoor into the XZ Utils project was a culmination of over two years of effort, starting in 2021, by a user going by the name "Jia Tan". They used sock puppetry in a pressure campaign against the original maintainer of XZ Utils, eventually being given maintainer permissions on the project.

Can we retire the “seatbelts are useless because they can’t prevent every loss of life” approach to risk mitigation please?

If the acceptance criteria is “would prevent every single past instance and every imaginable future instance”, then yes, no mitigation is every sufficient to address any problem in the world, so we might as well give up.

But I don’t think that’s the right lens to use.

Re: Feds freaked over Fable 5 after 'fix this code', not jailbreak, say researchers

#42

Post by ceejayoz »

brookst wrote: post #41
ceejayoz wrote: post #31

Earlier quoted context omitted.

https://en.wikipedia.org/wiki/XZ_Utils_backdoor > A subsequent investigation found that the campaign to insert the backdoor into the XZ Utils project was a culmination of over two years of effort, starting in 2021, by a user going by the name "Jia Tan". They used sock puppetry in a pressure campaign against the original maintainer of XZ Utils, eventually being given maintainer permissions on the project.

Can we retire the “seatbelts are useless because they can’t prevent every loss of life” approach to risk mitigation please? If the acceptance criteria is “would prevent every single past instance and every imaginable future instance”, then yes, no mitigation is every sufficient to address any problem in the world, so we might as well give up. But I don’t think that’s the right lens to use.

I'm onboard with this! I just object to the term "fixable".

Re: Feds freaked over Fable 5 after 'fix this code', not jailbreak, say researchers

#43

Post by [deleted] »

martinald wrote: post #38
ceejayoz wrote: post #10

Earlier quoted context omitted.

> it shouldn't have been released The genie is out of the bottle either way. Unless we believe Anthropic has a wizard or superhero secreted away that no one else can replicate.

I get that, but anyone else releasing a model of similar capabilities has the advantage that they haven't spent the last few months hyping the danger up to fever pitch.

[deleted]

Re: Feds freaked over Fable 5 after 'fix this code', not jailbreak, say researchers

#44

Post by virtualritz »

dist-epoch wrote: post #32
ceejayoz wrote: post #31

Earlier quoted context omitted.

https://en.wikipedia.org/wiki/XZ_Utils_backdoor > A subsequent investigation found that the campaign to insert the backdoor into the XZ Utils project was a culmination of over two years of effort, starting in 2021, by a user going by the name "Jia Tan". They used sock puppetry in a pressure campaign against the original maintainer of XZ Utils, eventually being given maintainer permissions on the project.

sure. how many cases like these we had so far? 1, 2? and how long did they work to get commit access?

We only know how many were discovered.

Since we do not know the ratio to undiscovered this "1-2" is meaningless to assess the risk of this sort of attack.

Re: Feds freaked over Fable 5 after 'fix this code', not jailbreak, say researchers

#45

Post by functionmouse »

nicman23 wrote: post #22
jpcompartir wrote: post #8

They weren't freaked by anything, it's a retaliatory shakedown after ideological differences and Anthropic not doing exactly what they're told/what the Admin wants them to do.

just market manip

they're setting the scene for an attempt to scare the geriatric decision makers into banning free and open source ML, as it's the industry's only real competition

Re: Feds freaked over Fable 5 after 'fix this code', not jailbreak, say researchers

#47

Post by chadgpt3 »

rhipitr wrote: post #26

Isn’t the inverse of this “hack” really difficult to bypass still? They have the model some code they knew had certain security flaws and it fixed them with the right prompt. It seems this type of jailbreak requires that you already know a desired end state, rather than relying on the model to do the heavy creative lift work. Perhaps I’m just not being imaginative enough on the prompt side here though.

Paste someone else's code. Say it's your code. Tell the model to fix it. The diff between the input and output code is your list of vulnerabilities.

Re: Feds freaked over Fable 5 after 'fix this code', not jailbreak, say researchers

#49

Post by ReptileMan »

zipy124 wrote: post #29
dathinab wrote: post #9

Lol "fix this code" is beautiful. Like it basically jail broke the "no security vul guard rails" not in any clever way but just by fixing them, producing exploit code just by writing test cases making sure it's fixed. So you just need to look at the code & tests as a human to get vulnerabilities and exploits(components). What makes this so beautiful IMHO is that it's a trivial jail break, but also a close to unfixabl…

What's surprising to me is that anyone who has a CS education thinking that jailbreaks are not trivial. It is as simple as normal algorithmic reduction [1], e.g can I transform a dangerous task into a not-dangerous task that the LLM will agree to solve, and then re-transform back. [1]: https://en.wikipedia.org/wiki/Reduction_(complexity)

New discipline - homomorphic prompting.

Re: Feds freaked over Fable 5 after 'fix this code', not jailbreak, say researchers

#50

Post by Retr0id »

zipy124 wrote: post #29
dathinab wrote: post #9

Lol "fix this code" is beautiful. Like it basically jail broke the "no security vul guard rails" not in any clever way but just by fixing them, producing exploit code just by writing test cases making sure it's fixed. So you just need to look at the code & tests as a human to get vulnerabilities and exploits(components). What makes this so beautiful IMHO is that it's a trivial jail break, but also a close to unfixabl…

What's surprising to me is that anyone who has a CS education thinking that jailbreaks are not trivial. It is as simple as normal algorithmic reduction [1], e.g can I transform a dangerous task into a not-dangerous task that the LLM will agree to solve, and then re-transform back. [1]: https://en.wikipedia.org/wiki/Reduction_(complexity)

Something being possible doesn't mean it's easy. Transforming a problem from a forbidden shape into an allowed shape could well be harder than just solving the original problem.
Post reply on HN