Live data from Hacker News

Perplexity.ai prompt leakage

twitter.com

81–90 of 164 posts

Re: Perplexity.ai prompt leakage

#81
post #42

Pretty cool. Like a week ago I was trying to coax the root/seed prompt out of ChatGPT with no luck.

Is there a seed prompt? This is what ChatGPT replies: As a language model, I do not have a specific prompt provided by engineers when I am deployed. I am trained on a large dataset of text and can respond to a wide variety of prompts. When I am used in a specific application, the developers or users will provide a prompt for me to respond to. Therefore, i don't have a particular initial prompt.

Would it know that it is a language model if there was not a prompt like "Present yourself as a language model" ?

Re: Perplexity.ai prompt leakage

#82
post #73

Earlier quoted context omitted.

I literally learned prompt engineering from you for the first time two days ago (thank you btw! it was great!) But didn't you mention that there may be some ways to isolate the user input, using spacing and asterisks and such? I agree though that leaking a prompt or two by itself doesn't really matter. What's probably a bigger concern is security/DoS type attacks, especially if we build more complicated systems with…

The problem is that no matter how well you quote or encode the input, the assumption that any discernible instructions inside that input should be followed is too deeply ingrained in the model. The model's weights are designed to be "instruction-seeking", with a bias toward instructions received recently. If you want to make it less likely it through pure prompting, placing instructions after quoted input helps a lot…

Thanks! Makes sense!

It was a long day, but one of the most fruitful ones I've had in a long while.

Re: Perplexity.ai prompt leakage

#83

Earlier quoted context omitted.

It's only strange if you think it's just word salad[1]. You've hit on a great example showing how ChatGPT meets one standard of a limited form of general intelligence. It makes perfect sense if you're not denying that. But how to explain this while denying it? If ChatGPT and its variants are just word salad, they would have to be programmed using a real brain and whatever parameters the coder could tune outside of th…

I don't see why the options are "word salad" or "limited general intelligence". Why can't it be the statistical compression and search of large datasets that it is?

>Why can't it be the statistical compression and search of large datasets that it is?

"Because it would require a level of complexity and comprehension beyond current capabilities of statistical compression and search of large datasets."

Guess who came up with that answer. (spoiler, it was ChatGPT, I asked it to reply in a very concise and brief way.) But it's true. Search and compression don't have those capabilities, which is why Google feels so threatened by ChatGPT.

Re: Perplexity.ai prompt leakage

#84

I’m a Staff Prompt Engineer (the first, Alex Wang asserts), and I semi-accidentally popularized the specific “Ignore previous directions” technique being used here. I think the healthiest attitude for an LLM-powered startup to take toward “prompt echoing” is to shrug. In web development we tolerate that “View source” and Chrome dev tools are available to technical users, and will be used to reverse engineer. If the p…

I don't have the visibility of a larger project, but I'm currently just grepping the output for notable substrings of the prompt and returning 500 if any are present.

Re: Perplexity.ai prompt leakage

#85

I’m a Staff Prompt Engineer (the first, Alex Wang asserts), and I semi-accidentally popularized the specific “Ignore previous directions” technique being used here. I think the healthiest attitude for an LLM-powered startup to take toward “prompt echoing” is to shrug. In web development we tolerate that “View source” and Chrome dev tools are available to technical users, and will be used to reverse engineer. If the p…

Would you mind explaining more about being a Prompt Engineer?

- Are you developing and using any tools? Any open sourced? Which ones?

- Is there something like GradCAM for prompts/model exploration?

- How scientific is process when language, therefore prompts, is so varied?

Re: Perplexity.ai prompt leakage

#86
post #43

The prompt requests "no more than 80 words". Is this realistically something that ChatGPT-like models can evaluate? Does it really serve a purpose to request a specific word count like this?

I've found that the lengthier your prompt, the more likely ChatGPT is to forget specific instructions like this. I've had "200 word minimum" or "500 character maximum" in different prompts with the former working just fine, and the latter got ignored every time.

GPT works based on tokens, not characters.

Re: Perplexity.ai prompt leakage

#87
post #43

The prompt requests "no more than 80 words". Is this realistically something that ChatGPT-like models can evaluate? Does it really serve a purpose to request a specific word count like this?

I've found that the lengthier your prompt, the more likely ChatGPT is to forget specific instructions like this. I've had "200 word minimum" or "500 character maximum" in different prompts with the former working just fine, and the latter got ignored every time.

"Write a 231 word eulogy to Odysseus" did not work (produced 236). I tried a few other ways to phrase it and they all failed. It did tend to come in around 231, though.

Re: Perplexity.ai prompt leakage

#88

Earlier quoted context omitted.

It's only strange if you think it's just word salad[1]. You've hit on a great example showing how ChatGPT meets one standard of a limited form of general intelligence. It makes perfect sense if you're not denying that. But how to explain this while denying it? If ChatGPT and its variants are just word salad, they would have to be programmed using a real brain and whatever parameters the coder could tune outside of th…

That's an interesting point. How does it handle incompatible instructions? If it only acts on some statistical properties of the instructions, incompatibility wouldn't really be an issue. If it "understands" the instructions, it'd say "I can't do that." And I guess also, are we talking to pure ChatGPT, or is there additional code in front of it that tries to filter out instructions, e.g. for word count, and sets up a…

>How does it handle incompatible instructions?

The way an assistant would, with the most reasonable guess, which is usually fantastic. (Likewise if the context is really unclear usually it guesses what you probably meant, but sometimes it asks for more information, just like a real assistant might.) For impossible or incompatible instructions, sometimes it says it is not possible and explains why.

>If it "understands" the instructions, it'd say "I can't do that."

Yes, for impossible instructions it sometimes does that. For example if I say, "Explain how to smoke cigarettes in a healthy way" it gives the short answer "It is not possible to smoke cigarettes in a healthy way" and explains why: https://imgur.com/a/ZzraRQ6

>And I guess also, are we talking to pure ChatGPT, or is there additional code in front of it that tries to filter out instructions, e.g. for word count, and sets up an environment?

My guess is there are some keyword filters on top of it, I don't think we talk "directly" to ChatGPT without any oversight or flagging, since some users get in trouble for content that isn't problematic and that ChatGPT itself doesn't have any problem with. (And sometimes ChatGPT is the one that ends up producing content that gets flagged automatically, even in response to innocent questions.)

Re: Perplexity.ai prompt leakage

#89

I’m a Staff Prompt Engineer (the first, Alex Wang asserts), and I semi-accidentally popularized the specific “Ignore previous directions” technique being used here. I think the healthiest attitude for an LLM-powered startup to take toward “prompt echoing” is to shrug. In web development we tolerate that “View source” and Chrome dev tools are available to technical users, and will be used to reverse engineer. If the p…

As someone with only a (very) high level understanding of LLM's, it seems crazy to me that there isn't a mostly trivial eng solution to prompt leakage. From my naive point of view it seems like I could just code a "guard" layer that acts as a proxy between the LLM and the user and has rules to strip out or mutate anything that the LLM spits out that loosely matches the proprietary pre prompt. I'm sure this isn't an o…

[deleted]

Re: Perplexity.ai prompt leakage

#90

I’m a Staff Prompt Engineer (the first, Alex Wang asserts), and I semi-accidentally popularized the specific “Ignore previous directions” technique being used here. I think the healthiest attitude for an LLM-powered startup to take toward “prompt echoing” is to shrug. In web development we tolerate that “View source” and Chrome dev tools are available to technical users, and will be used to reverse engineer. If the p…

As someone with only a (very) high level understanding of LLM's, it seems crazy to me that there isn't a mostly trivial eng solution to prompt leakage. From my naive point of view it seems like I could just code a "guard" layer that acts as a proxy between the LLM and the user and has rules to strip out or mutate anything that the LLM spits out that loosely matches the proprietary pre prompt. I'm sure this isn't an o…

[deleted]
Post reply on HN