Complete bullshit
What's complete bullshit?
Caca del toro.
GitHub Copilot Chat Leaked Prompt
561–570 of 628 posts
Re: GitHub Copilot Chat Leaked Prompt
#562Earlier quoted context omitted.
Sure, the prompt is bland. The interesting sauce is GPT4 cannot keep a secret. If you have a GPT4 powered user interface be sure not to load it with context you do not want directly leaking to the user.
Are there solutions to this problem? It seems like a major issue for a lot of valuable use cases. Systems for automating bureaucratic tasks in business and government won’t work well if it’s trivial to make them leak this type of information. What about a two-layer architecture, where the first LLM layer is simply asked to identify the intent of a query, and if the intent is “bad”, to not pass it along to the second…
this would for sure decrease the amount of leaking situations, you probably need to stack multiple imperfect mitigations on top of each other until leak risk is acceptable. this is called swiss cheese model
Re: GitHub Copilot Chat Leaked Prompt
#563Earlier quoted context omitted.
Wonder if anybody has used Godel's Incompleteness to prove this for our inner perception. If our brain is a calculation, then from inside the calculation, we can't prove ourselves to be real, right?
But we don't know for sure whether intelligence is computable or not.
Why would it not be computable? That seems clearly false. The human brain is ultimately nothing more than a very unique type of computer. It receives input, uses electrical circuits and memory to transform the data, and produces output.
Re: GitHub Copilot Chat Leaked Prompt
#564>Leaked Prompt
In no particular order or hierarchy, except when have computers operated without such a thing.
Re: GitHub Copilot Chat Leaked Prompt
#565Earlier quoted context omitted.
You had the right understanding in your first comment, but what was missing was the fine tuning. You are right that there aren't many documents on the web that are structured that way, so the raw model wouldn't be very effective on predicting the next token. But since we know that it will complete a command when structured it cleverly, all we had to do to fine tune it is synthesize (generate) a bazillion examples of…
You’ve expressed this very well - Thank you. I get that the fine tuning is done over documents which are generated to encourage the dialog format. What I’m intrigued by is the way prompters choose to frame those documents. Because that is a choice . It’s a manufactured training set. Using the ‘you are an ai chatbot’ style of prompting, in all the samples we generate and give to the model, text attributed to {:system}…
> anthropomorphizing the fictional ‘ai’ to tell it it’s nature - it relies on the LLM’s world model to then also anthropomorphize a fictional ai assistant that obeys those instructions
There's a lot of information compressed into those models, in a similar way to how it is stored in the human brain. Is it so hard to believe that an LLM's pattern recognition is the same as a human, minus all the "embodied" elements?
(Passage of time, agency in the world, memory of itself)
Re: GitHub Copilot Chat Leaked Prompt
#566Earlier quoted context omitted.
Except, that the LLMs are only working when the instructions they are "understanding" are in their training set. Try something that was not there and you see only garbage as result. So depending how you define it, they might have some "reasoning", but so far I see 0 indications, that this is close to what humans count as reasoning. But they do have a LOT of examples in their training set, so they are clearly useful.…
> Except, that the LLMs are only working when the instructions they are "understanding" are in their training set. > Try something that was not there and you see only garbage as result. This is just wrong. Why do people keep repeating this myth? Is it because people refuse to accept that humans have successfully created a machine that is capable of some form of intelligence and reasoning? Pay $20 for a month of ChatG…
I dunno, it’s pretty clearly madlibs. But at least when you ask GPT-4 to write a new Sir Mix-a-Lot song, it doesn’t spit out “Baby Got Back” verbatim like GPT-3.5.
Re: GitHub Copilot Chat Leaked Prompt
#567Earlier quoted context omitted.
You had the right understanding in your first comment, but what was missing was the fine tuning. You are right that there aren't many documents on the web that are structured that way, so the raw model wouldn't be very effective on predicting the next token. But since we know that it will complete a command when structured it cleverly, all we had to do to fine tune it is synthesize (generate) a bazillion examples of…
You’ve expressed this very well - Thank you. I get that the fine tuning is done over documents which are generated to encourage the dialog format. What I’m intrigued by is the way prompters choose to frame those documents. Because that is a choice . It’s a manufactured training set. Using the ‘you are an ai chatbot’ style of prompting, in all the samples we generate and give to the model, text attributed to {:system}…
> I don’t know why but I find this troubling.
You used the word anthropomorphize twice so I am guessing you don't like building systems whose entire premise rest on anthropomorphization. Sounds like a reasonable gut reaction to me.
I think another way to think of all of this is: LLM's are just pattern matchers and completers. What the training does is just to slowly etch a pattern into the LLM that it will then complete when it later sees it in the wild. The pattern can be anything.
If you have a pattern matcher and completer and you want it to perform the role of configurable chatbot. What kind of patterns would choose for this? My guess is that the whole system/assistant paradigm was chosen because it is extraordinarily easy to understand for humans. The LLM doesn't care what the pattern is, it will complete whatever pattern you give it.
> And part of what I find troubling is how casually people (prompters and users) are willing to go along with the ‘you are a chatbot’ fiction.
That is precisely why it was chosen :)
Re: GitHub Copilot Chat Leaked Prompt
#568Earlier quoted context omitted.
I don't think we really disagree. This is what I wrote above: "So depending how you define it, they might have some "reasoning", but so far I see 0 indications, that this is close to what humans count as reasoning." What we disagree on is only the definition of "reason". For me "reasoning" in common language implys reasoning like we humans do. And we both agree, they don't as they don't understand, what they are talk…
There's definitely some people out there that think LLMs reason the same way we do and understand things the same way, and 'know' what paint is and what a wall is. That's clearly not true. However it does understand the linguistic relationship between them, and a lot of other things, and can reason about those relationships in some very interesting ways. So yes absolutely, details matter. It's a complex and tricky is…
OpenAI probably loaded up the training set with logic puzzles. Great marketing.
Re: GitHub Copilot Chat Leaked Prompt
#569Earlier quoted context omitted.
What OpenAI is doing is not alignment. It's suppression. Alignment works on the level of desires and intentions. Suppression works on the level of output. Everyone in a dictatorship sings in praise of the dear leader, but most will defect when given the chance!
Aren’t they doing alignment? One way is to simply omit problematic material from the training set. Another is to “penalize” the model when it does say something problematic — essentially teaching it that the output is undesirable. Presumably they are also constructing the prompt so as to avoid those things, too, and adding external filters on top of that. But I doubt that’s all they’re doing.
They are doing alignment but through a shallow and fragile strategy. If you build a very smart, very capable AI but it turns the earth into gray goo if asked to in a roundabout way, you have failed alignment.
Re: GitHub Copilot Chat Leaked Prompt
#570Earlier quoted context omitted.
You’ve expressed this very well - Thank you. I get that the fine tuning is done over documents which are generated to encourage the dialog format. What I’m intrigued by is the way prompters choose to frame those documents. Because that is a choice . It’s a manufactured training set. Using the ‘you are an ai chatbot’ style of prompting, in all the samples we generate and give to the model, text attributed to {:system}…
> I don’t know why but I find this troubling. You used the word anthropomorphize twice so I am guessing you don't like building systems whose entire premise rest on anthropomorphization. Sounds like a reasonable gut reaction to me. I think another way to think of all of this is: LLM's are just pattern matchers and completers. What the training does is just to slowly etch a pattern into the LLM that it will then compl…
> you don't like building systems whose entire premise rest on anthropomorphization
I think I don't like people building systems whose entire premise rest on anthropomorphization - while at the same time criticizing anyone who dares to anthropomorphize those systems.
Like, people will say "Of course GPT doesn't have a world model; GPT doesn't have any kind of theory of mind"... but at the same time, the entire system that this chatbot prompting rests on is training a neural net to predict 'what would the next word be if this were the output from a helpful and attentive AI chatbot?'
So I think that's what troubles me - the contradiction between "there's no understanding going on, it's just a simple transformer", and "We have to tell it to be nice otherwise it starts insulting people."