GitHub Copilot Chat Leaked Prompt

Post by wll » Sat, May 13, 2023, 1:43 PM UTC

wll wrote: post #446

simonw wrote: post #433

Earlier quoted context omitted.

I don't think "works well enough" is good enough. We are currently starting to wire LLMs up as AI-enhanced personal assistants - with the goal of giving them access to our email, and the ability to take actions on our behalf. If we widely deploy these systems the incentives for attackers to figure out prompt injection attacks that get last any probability-based filters we are using will be enormous. An attacker only…

How do we determine how vulnerable a system is without seeing how it is implemented? That is, can you generalize LLM usage to all apps and determine that the entire field is exposed? Obviously it is a problem for end users, but that’s more of an optics and trust angle, just as browser extensions are a privacy nightmare. I am as worried as you are on this front, given that people appear to be using browser extensions…

Post by frickinadorable » Sat, May 13, 2023, 1:45 PM UTC

jameshart wrote: post #437

tinco wrote: post #233

Earlier quoted context omitted.

You had the right understanding in your first comment, but what was missing was the fine tuning. You are right that there aren't many documents on the web that are structured that way, so the raw model wouldn't be very effective on predicting the next token. But since we know that it will complete a command when structured it cleverly, all we had to do to fine tune it is synthesize (generate) a bazillion examples of…

You’ve expressed this very well - Thank you. I get that the fine tuning is done over documents which are generated to encourage the dialog format. What I’m intrigued by is the way prompters choose to frame those documents. Because that is a choice . It’s a manufactured training set. Using the ‘you are an ai chatbot’ style of prompting, in all the samples we generate and give to the model, text attributed to {:system}…

Post by Sharlin » Sat, May 13, 2023, 1:55 PM UTC

adriand wrote: post #458

htag wrote: post #117

Earlier quoted context omitted.

Sure, the prompt is bland. The interesting sauce is GPT4 cannot keep a secret. If you have a GPT4 powered user interface be sure not to load it with context you do not want directly leaking to the user.

Are there solutions to this problem? It seems like a major issue for a lot of valuable use cases. Systems for automating bureaucratic tasks in business and government won’t work well if it’s trivial to make them leak this type of information. What about a two-layer architecture, where the first LLM layer is simply asked to identify the intent of a query, and if the intent is “bad”, to not pass it along to the second…

Post by simonw » Sat, May 13, 2023, 1:59 PM UTC

brap wrote: post #453

MagicMoonlight wrote: post #429

I like that none of these "rules" actually mean anything because it's just a language model. It will follow them if it thinks following them is the most probable next sequence of characters (although it might not follow them because its output has randomness rather than always being the most probable next characters) but that's not an actual guarantee. There's no thinking structure. It needs an actual brain system wh…

I wonder if this can be solved by adding a disconnected, 2nd layer which can answer meta-questions about the bot’s answers, such as “does this answer follow the rules?”, and this can be used to tweak the answers. Perhaps it can also be used to answer questions that require some sort of chain-of-thought, like “what would be the first step to answering the question? What is the next one?”, etc, then compile these “thou…

Post by hgsgm » Sat, May 13, 2023, 2:06 PM UTC

FrustratedMonky wrote: post #326

iforgotpassword wrote: post #303

Earlier quoted context omitted.

No you cannot know this, because you might just be simulating that you understand. You cannot reliably observe a system from within itself. It's like running an antivirus on an infected system is inherently flawed, because there might be some malware running that knows every technique the antivirus uses to scan the system and can successfully manipulate every one of them to make the system appear clean. There is no g…

Wonder if anybody has used Godel's Incompleteness to prove this for our inner perception. If our brain is a calculation, then from inside the calculation, we can't prove ourselves to be real, right?