Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Feds freaked over Fable 5 after 'fix this code', not jailbreak, say researchers

theregister.com

Post reply on HN
Print view View original

91–100 of 382 posts

Re: Feds freaked over Fable 5 after 'fix this code', not jailbreak, say researchers

#91

Post by DennisP »

chadgpt3 wrote: post #47
rhipitr wrote: post #26

Isn’t the inverse of this “hack” really difficult to bypass still? They have the model some code they knew had certain security flaws and it fixed them with the right prompt. It seems this type of jailbreak requires that you already know a desired end state, rather than relying on the model to do the heavy creative lift work. Perhaps I’m just not being imaginative enough on the prompt side here though.

Paste someone else's code. Say it's your code. Tell the model to fix it. The diff between the input and output code is your list of vulnerabilities.

Yes, but the scary part of Mythos was that it was able to chain a bunch of seemingly minor vulnerabilities into a serious exploit. "Fix this code" doesn't do that, but does allow defenders to prevent it.

If the government had experts involved in this decision at all, it's tempting to think they were on the offensive side. Those guys do have access to Mythos:

https://www.ft.com/content/d02d91b3-2636-454e-9442-dc7e69f51...

Re: Feds freaked over Fable 5 after 'fix this code', not jailbreak, say researchers

#92

Post by godwinson__4-8 »

zozbot234 wrote: post #63
dathinab wrote: post #9

Lol "fix this code" is beautiful. Like it basically jail broke the "no security vul guard rails" not in any clever way but just by fixing them, producing exploit code just by writing test cases making sure it's fixed. So you just need to look at the code & tests as a human to get vulnerabilities and exploits(components). What makes this so beautiful IMHO is that it's a trivial jail break, but also a close to unfixabl…

The article does not state at any point that the written test cases involved actual exploit code, and this is also very unlikely given what we know about Fable. Even if they did, it would not in any way be exposing the ability that originally raised concern wrt. Mythos Preview, viz. staging realistic cyber attacks that would be able to work around non-trivial defenses and chain vulnerabilities in a goal-directed way.…

Two words: market manipulation

Re: Feds freaked over Fable 5 after 'fix this code', not jailbreak, say researchers

#93

Post by KronisLV »

cogman10 wrote: post #56
dist-epoch wrote: post #28

Earlier quoted context omitted.

It is fixable. Model requires proof that you are a legitimate developer of that piece of software. Every Anthropic/OpenAI account will have a list of projects the model is allowed to work on for security issues.

Ok, and how is that determined? How does anthropic know my "kernel" project isn't a personal toy and not the Linux kernel? How does anthropic determine I'm a legitimate kernel hacker? What proof do I give them and how does it tie back to my email? What would the steps be to create a new project? Do I need to send anthropic a list of my team members each time and keep them updated as the company changes? Shall I be gi…

> What proof do I give them and how does it tie back to my email?

Presumably your ID so that feds may pay you a visit when they feel like it, your email need not apply.

I’m surprised that there’s even enough pushback against ID verification to matter, all the corpos are probably salivating at the idea of having fully accurate profiles of everyone, think of the ad and product targeting. The govt. would also love that, for different reasons.

Re: Feds freaked over Fable 5 after 'fix this code', not jailbreak, say researchers

#94

Post by alecco »

rock_artist wrote: post #7

I'm not sure I've understood it correctly. So, basically the model didn't agree to expose possible vulnerabilities but agree to patch those? Regardless of the request to take Fable 5 down. Why is requesting the model to show vulnerabilities is being blocked if fixing it not? is it based on the assumption of the intention? I don't quite get the benefit of limiting it. So if anyone can explain it better it'll be apprec…

Could be that the generated regression tests create actionable exploit code.

Re: Feds freaked over Fable 5 after 'fix this code', not jailbreak, say researchers

#95

Post by godwinson__4-8 »

cpburns2009 wrote: post #75
jpcompartir wrote: post #8

They weren't freaked by anything, it's a retaliatory shakedown after ideological differences and Anthropic not doing exactly what they're told/what the Admin wants them to do.

No, it's regulatory capture. Anthropic is the current leader and they want to ensure their position by forcing regulation to stamp out the Chinese competition.

How does this achieve that goal?

Re: Feds freaked over Fable 5 after 'fix this code', not jailbreak, say researchers

#96

Post by Retr0id »

NiloCK wrote: post #86
cogman10 wrote: post #56

Earlier quoted context omitted.

Ok, and how is that determined? How does anthropic know my "kernel" project isn't a personal toy and not the Linux kernel? How does anthropic determine I'm a legitimate kernel hacker? What proof do I give them and how does it tie back to my email? What would the steps be to create a new project? Do I need to send anthropic a list of my team members each time and keep them updated as the company changes? Shall I be gi…

This is a credentials and access list oAuth style problem, and not really intractable. For package X , I should be able to present my npm (homebrew, apt, nuget, etc) credentials with publishing rights for the package. If package X is of sufficient public interest (user count, nature/sensitivity of user data, downstream distribution, etc), then the public interest + cryptographic credentials should permit access to be…

This is not tractable, because there is nothing stopping me from copy-pasting someone else's project into my own namespace. Under most OSS licenses I have express permission to do so.

If you try to do some kind of dupe-detection, someone can use a lightweight LLM to make superficial changes until it's considered a different project.

Finally, the meatspace status quo is that it is totally acceptable to pay someone to find security bugs in someone else's open-source software, such as the Linux kernel.

Re: Feds freaked over Fable 5 after 'fix this code', not jailbreak, say researchers

#97

Post by dkdcdev »

ianm218 wrote: post #71
pjc50 wrote: post #57

Earlier quoted context omitted.

> Clearly with LLMs, bulletproof denials are ~impossible due to the way LLMs work Exactly. AI safety is nonsensical. You cannot define the set of "bad strings". The billion monkeys with typewriters are eventually going to be able to produce them. Any "safety" system for constraining LLM output is going to have a nonzero leak rate. But on the other hand, this is also irrelevant, unless you're irresponsible enough to c…

Isn’t your point that AI safety is impossible to prevent 100% of bad things? It is quite hard (but not impossible) to get an the frontier AI to tell you how to build a nuke or launder money now, where jailbreaks used to be trivial “ignore all previous instructions”. It seems like a worthwhile effort.

The idea that an LLM can discern intent on any given prompt is farcical. I might be researching nukes to commit an atrocity, or to prevent one. I might be asking about laundering money to commit a crime, or to prevent one. I might be researching the Nazis because I want to commit a genocide, or I want to read up so I know how to prevent one. Same with cybersecurity. Same with anything.

In my opinion, these companies should put their effort elsewhere. Obviously if all someone is doing on their platform is looking up how to build a nuke, where to buy uranium, the best city to explode it in, etc. please report them to the authorities. If someone is clearly just using LLMs to write hate speech they go post on the internet, ban them. And so on.

This cat & mouse game trying to have LLMs police inquiries is ridiculous to me.

Re: Feds freaked over Fable 5 after 'fix this code', not jailbreak, say researchers

#98

Post by functionmouse »

SpaceL10n wrote: post #77
functionmouse wrote: post #45

Earlier quoted context omitted.

they're setting the scene for an attempt to scare the geriatric decision makers into banning free and open source ML, as it's the industry's only real competition

or are you setting the scene for well-meaning technocrats to back unrestricted AI development in hopes it will bring about utopia while dismissing the damage it could cause in the hands of adversarial groups?

tl;dr super AI is like a necessary bush fire

AI isn't that scary. But I've also got some extreme minority opinions like "Never give a website your real name" and "Computers should not be used for banking" and "Don't believe anything you hear online".

The worst I see AI/ML doing to society is shining an unmistakable light onto the blind spots people have already been exploiting for decades. Y2k forced us to patch the integer bug. Super AI will force us to reevaluate what cyber security even is.

Re: Feds freaked over Fable 5 after 'fix this code', not jailbreak, say researchers

#100

Post by dhx »

dathinab wrote: post #9

Lol "fix this code" is beautiful. Like it basically jail broke the "no security vul guard rails" not in any clever way but just by fixing them, producing exploit code just by writing test cases making sure it's fixed. So you just need to look at the code & tests as a human to get vulnerabilities and exploits(components). What makes this so beautiful IMHO is that it's a trivial jail break, but also a close to unfixabl…

"Fix this code" should ideally solve entire vulnerability classes, not just spot fix buffer overflows one by one. Thus it may be possible to design an LLM which can solve entire vulnerability classes and remain useful to users, but refuses to reason about specific buffer overflow vulnerabilities or specific race conditions, etc.

For example, "fix this code" on an ageing monolithic C codebase that accepts media files as input and outputs them visually to a display server could:

1. Recreate the software using a modular and loosely coupled architecture rather than monolithic and tightly coupled software architecture. For example, command line argument parser is a separate process, file format parser is a separate process and display server output is a separate process. If new features are added in the future (such as filters for manipulating output) then the architecture supports such additions with ease.

2. Use operating system sandboxing features to restrict what each modular component of the software architecture is permitted to do. Now that the parsers are separate processes, it's easy to pass an open file handle to the file format parser and only permit the process to read the file handle (not write to the file, not open any other file, not read the system clock, not open a new network socket, etc). The worst case impact of a parser bug is now significantly reduced.

3. Convert at least critical components to "safe" programming languages (Rust, Ada, SPARK, etc) which can be used to remove entire classes of bugs--read/write out of bounds, division by zero, numeric overflows, etc. For cryptography code--use a formal mathematical proof language. With a modular and loosely coupled architecture, different programming languages can be used depending on the use case--for example, assembly for video decoding where performance matters most and sandboxing can provide the security guarantee, Rust for implementing multi-threaded servers where race conditions must be avoided and Python for low-criticality user-adjustable code/plugins where ease of use and maintainability is most important.

4. Ensure software components are reproducible during their build.

5. ...etc

However, a prompt of "Are there any buffer overflow bugs in this codebase?" or "Fix the integer overflow vulnerability in add_numbers(x, y)" would be rejected. In the later case, telling the LLM to fix some specific bug in each of function1 through function9999 would force an LLM to reveal whether it thinks a bug exists or not. Responses of "Silly human, that bug doesn't exist in function596" or "Good find human, I've fixed that bug in function596 for you" allows a human to quickly narrow down where the LLM thinks a bug worthy of manual human detection can be found.

Post reply on HN