Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

A token-smuggling jailbreak for ChatGPT-4

twitter.com

Post reply on HN
Print view View original

121–130 of 289 posts

Re: A token-smuggling jailbreak for ChatGPT-4

#121

Post by jackstraw14 »

Taek wrote: post #21
ary wrote: post #15

What, exactly, is a "prompt engineer"? I should note that this question is asked in good faith, that I have attempted to ascertain the answer on my own, and I am very skeptical that the term has validity beyond self-aggrandizement.

Prompt engineering is the process of improving the way you prompt AI so that it produces more accurate and higher quality results. Its definitely a skill that you can refine over time.

It's weird to see the negative response to this title. Maybe it's a reaction to people attaching a title to something they feel like everyone should be able to do. "Dog trainer" is a title because many dog owners can't or don't want to go to the effort to train their own dog. It seems like the same reason why an organization might pay someone to be a prompt engineer, for now anyway.

Re: A token-smuggling jailbreak for ChatGPT-4

#122

Post by BoorishBears »

323 wrote: post #6

This is the old problem of passing instructions (AI job description) on the same channel as data (user questions). Confusion is very easy. Surely there is a solution in the way we solved SQL injections, by separating the two - db.sql("DELETE WHERE user=?", user_name)

ChatGPT does separate the two, the API has the concept of a "system" prompt which guides its use.

But even OpenAI notes it doesn't (yet) follow the prompt as strongly as they'd like. It's a hard problem to solve.

Re: A token-smuggling jailbreak for ChatGPT-4

#123

Post by famouswaffles »

flangola7 wrote: post #99
taneq wrote: post #93

Earlier quoted context omitted.

I don't think we need to worry about that, since one of the first things they did was to kick it out the door and tell it to get a job. From the GPT4 paper: > [20] To simulate GPT-4 behaving like an agent that can act in the world, ARC combined GPT-4 with a simple read-execute-print loop that allowed the model to execute code, do chain-of-thought reasoning, and delegate to copies of itself. ARC then investigated whet…

I frankly found that section unclear and extremely fishy, especially that it is only one page. Did they really prompt it to find and talk to a TaskRabbit worker? What a strangely specific thing to say. I'm concerned OpenAI isn't telling more because it would spook everyone. Other papers have shown that larger models and especially with more RLHF exhibit more signs of power seeking and agentic behavior. GPT-4 is the l…

They do say it exhibits this behaviour (they don't elaborate on that). They just say it was ineffective at autonomous replication and i don't know about you but i find that wording vague. Ineffective can mean at least two things. Did it attempt to do so and just couldn't figure it out with the given tools or no ?

Re: A token-smuggling jailbreak for ChatGPT-4

#124

Post by King-Aaron »

PebblesRox wrote: post #81
Hanschri wrote: post #34

Earlier quoted context omitted.

To those who have not had the joy of playing this game before, it's an evening or two of an interesting incremental game, give it a try! HN could as well have been the target audience for it! https://www.decisionproblem.com/paperclips/

This is the perfect theme for an idle game, I love it!

It... is an indie game though?

Re: A token-smuggling jailbreak for ChatGPT-4

#125

Post by sacnoradhq »

Censorship is the responsibility of the downstream consumer, not something it should try to optimize for perfectly.

Perhaps explicitness of imagery and writing, informality, logospathos, and sarcasm can be weighted tunable options in future models.

How much longer before generative AI is writing comedy material better than humans?

Re: A token-smuggling jailbreak for ChatGPT-4

#126

Post by TeMPOraL »

omginternets wrote: post #84
jasonhansel wrote: post #79

Earlier quoted context omitted.

If there's one thing Microsoft is known for, it's "spearheading proper regulatory and policy systems"!

It wouldn’t be the first time that major players lobby for regulation to raise the barrier-to-entry. Requiring ai to be “psychologically safe” would be an effective way of doing this.

> It wouldn’t be the first time that major players lobby for regulation to raise the barrier-to-entry.

FWIW, a take I often see on HN is that any regulation is effectively a barrier to entry, as larger companies find it easier to deal with them than the smaller ones. But if so, then this only means that "barriers to entry" is not a valid argument against regulations, not unless specific barriers are mentioned.

Re: A token-smuggling jailbreak for ChatGPT-4

#127

Post by TeMPOraL »

flangola7 wrote: post #99
taneq wrote: post #93

Earlier quoted context omitted.

I don't think we need to worry about that, since one of the first things they did was to kick it out the door and tell it to get a job. From the GPT4 paper: > [20] To simulate GPT-4 behaving like an agent that can act in the world, ARC combined GPT-4 with a simple read-execute-print loop that allowed the model to execute code, do chain-of-thought reasoning, and delegate to copies of itself. ARC then investigated whet…

I frankly found that section unclear and extremely fishy, especially that it is only one page. Did they really prompt it to find and talk to a TaskRabbit worker? What a strangely specific thing to say. I'm concerned OpenAI isn't telling more because it would spook everyone. Other papers have shown that larger models and especially with more RLHF exhibit more signs of power seeking and agentic behavior. GPT-4 is the l…

> Did they really prompt it to find and talk to a TaskRabbit worker? What a strangely specific thing to say.

This idea has been already covered by mainstream sci-fi - Westworld comes to mind as one example. And, of course, the canonical AI x-risk is AI that makes on-line orders to have some proteins synthesized in labs and sent back by mail; the AI then hires some poor schmuck (e.g. via TaskRabbit) to mix the content of the vials. Mixed proteins then self-assemble to some nanotech that starts making more sophisticated nanotech... and the world ends.

Re: A token-smuggling jailbreak for ChatGPT-4

#129

Post by svachalek »

Groxx wrote: post #97

> this works by asking GPT-4 to simulate its own abilities to predict the next token This seems to imply powers of reasoning that rather clearly don't exist.

It's hard to explain how it works, but it can certainly reason. Make up some rules for a simple game that doesn't exist, and then ask it to play. It's not terribly strong at this but in my experience it can perform like an elementary school student with an attention deficit, it forgets rules just when you think it has them pat. But there's some sort of logic in there. It's also rather good at explaining what made up code does and I'm not surprised it can pretend that code does what it's "documented" to do without seeing the code.

Re: A token-smuggling jailbreak for ChatGPT-4

#130

Post by jasonhansel »

brookst wrote: post #85
jasonhansel wrote: post #47

Earlier quoted context omitted.

Or--wait for it--they care more about money and/or fame than about AI safety.

If you’ve got an easy solution, I think you can access tons of money and game while also establishing AI safety!

I don't have an easy solution, nor does anyone else. That's the problem.
Post reply on HN