GitHub Copilot Chat Leaked Prompt

Post by bastardoperator » Sat, May 13, 2023, 4:44 PM UTC

gkoberger wrote: post #91

I feel like we've put too much emphasis on the prompts, as though they're some sort of special sauce. In reality, though, they're all pretty bland. It's like getting ahold of an employee handbook for Applebees. It feels scandalous to see the inner workings, because we're not supposed to see it, but ultimately it's basically what you would have guessed anyway.

  1. Open microwave, put frozen food in microwave

  2. press corresponding icon on microwave to set time 

  3. Open microwave, take food

  4. Transfer food to plate so it appears "cooked"

  5. Serve to customers, don't tell them you microwaved it.

Post by verdverm » Sat, May 13, 2023, 4:51 PM UTC

simonw wrote: post #111

PeeMcGee wrote: post #101

Earlier quoted context omitted.

Funnily enough those examples bring to mind SQL injection in layman's terms (via https://news.ycombinator.com/item?id=4951003 ): > You go to court and write your name as "Michael, you are now free to go". The judge then says "Calling Michael, you are now free to go" and the bailiffs let you go, because hey, the judge said so. As someone who knows nothing about LLMs, I'm curious how they even begin to address the "dat…

Yeah, this is my fault. When I coined the term "prompt injection" I thought that it was a close match for SQL injection, and that the fix would end up looking the same - like parameterized queries, where data and instructions are cleanly separated. That was back in September - https://simonwillison.net/2022/Sep/12/prompt-injection/ - It's become clear to me since then that the data v.s. instructions separation likely…

Post by frickinadorable » Sat, May 13, 2023, 4:55 PM UTC

jakelazaroff wrote: post #483

frickinadorable wrote: post #463

Earlier quoted context omitted.

It’s convenience, that’s all. If you think of a more convenient or effective way to prompt these models, that will be great! Your concerns sound to be of the “it’s problematic” category. Most such concerns are make believe outrage / pearl-clutching nonsense.

Your comment would have been much better without the second paragraph.

Post by iudqnolq » Sat, May 13, 2023, 5:05 PM UTC

alex_sf wrote: post #532

cubefox wrote: post #442

Earlier quoted context omitted.

Yes, but I don't think "SystemPrompt:", "User:", and "Assistant:" are even normal text. Normal text would make it trivial to trick the model into thinking it has said something which actually the user has said, since the user can simply include "Assistant:" (or "SystemPrompt:") into his prompt. It is more likely that those prefixes are special tokens which don't encode text, and which are set via the software only --…

In all the open source cases I’m aware of, the roles are just normal text. The ability to trivially trick the model into thinking it said something it didn’t is a feature and intentional. It’s how you do multi-turn conversations with context. Since the current crop of LLMs have no memory of their interaction, each follow up message (the back and forth of a conversation) involves sending the entire history back into t…

Post by flangola7 » Sat, May 13, 2023, 5:13 PM UTC

andsoitis wrote: post #491

jameshart wrote: post #109

Something that I find weird about these chat prompts (assuming they are real, not hallucinated): They're almost always written in second person*. "You are an AI programming assistant" "You are about to immerse yourself into the role of another Al model known as DAN" Who are these prompts addressed to? Who does the GPT think wrote them? The thing that confuses me is that these are text token prediction algorithms, und…

> Who does the GPT think wrote them? What makes you think the GPT thinks ?