Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Prompt Engineering: Steer a large pretrained language model to do what you want

lilianweng.github.io

Post reply on HN
Print view View original

41–50 of 53 posts

Re: Prompt Engineering: Steer a large pretrained language model to do what you want

#41

Post by crooked-v »

ravenstine wrote: post #35
eddsh1994 wrote: post #15

I wonder if "prompt engineering" will be replaced by improvements to LLMs within the next couple years and this is just a bad DX step gap along the journey

As long as language is used, prompt engineering will always be a thing. The term "prompt engineering" is just a fancy way of describing being a good communicator.

A good communicator, but also a good sophist. For example, I discovered pretty quick that I could twist the ChatGPT 4 guardrails into knots, even without "DAN" prompts, just by feeding it a lot of philosophical nonsense about semantics and meaning.

Re: Prompt Engineering: Steer a large pretrained language model to do what you want

#42

Post by Dr_Birdbrain »

dpflan wrote: post #19
Dr_Birdbrain wrote: post #13

Earlier quoted context omitted.

I spent a few years of my career in formal verification. As a result of that, I suspect that the desiderata of “technical alignment” research is most likely undecidable, and the existing RLHF and prompt engineering approaches are probably the best we can do. A neural network is a large, nonlinear function. In many applications, such as those that release their source code and allow fine tuning, it is adaptive. Intern…

Isn’t prompt engineering a human way to explore the latent space of the model to create the mathematical interactions we want/deem useful? A human way of traversing the hyperdimensionalty. Given your background, would you say that prompt engineering is about finding the spaces in latent space the bound the model’s “knowledge” for our uses?

Yes, but I think the folks who work on technical alignment would like to have stronger guarantees—I think they would like to end up with formal guarantees that the AI will only be helpful, and not harmful.

Re: Prompt Engineering: Steer a large pretrained language model to do what you want

#43

Post by null0pointer »

danpalmer wrote: post #10

I find it fascinating that emergent behaviour of these models is being used to wrangle them into doing what we want, but I find it worrying that this is considered the right approach for issues of safety. It seems obvious to me that an out-of-band approach to safety and security is necessary and that this method is never going to work in a generalised way.

I've been wondering this too. How can the smart people working on these systems think that this is the right way to control them? Jailbreaks and other ways of getting them to do unintended things seem akin to social engineering to me. Even well trained humans can be convinced to break protocol sometimes, and I don't think anyone would say these things have a superhuman ability to detect social engineering attempts yet.

Re: Prompt Engineering: Steer a large pretrained language model to do what you want

#44

Post by chankstein38 »

RyEgswuCsn wrote: post #8

I wonder how long it will take before people start to use LLMs to generate prompts...

I already use it to generate prompts for AI art generation! I'll have it randomly generate scenarios and describe them visually then frequently I'll feed it into another conversation I have that transforms them to something less human friendly and more SD friendly

Re: Prompt Engineering: Steer a large pretrained language model to do what you want

#45

Post by ahzhou »

eddsh1994 wrote: post #15

I wonder if "prompt engineering" will be replaced by improvements to LLMs within the next couple years and this is just a bad DX step gap along the journey

Advances seem to make some of the hacks less necessary (such as CoT behaving better with newlines), but precisely specifying what you want will always be useful.

Re: Prompt Engineering: Steer a large pretrained language model to do what you want

#46

Post by KRAKRISMOTT »

Dr_Birdbrain wrote: post #13
danpalmer wrote: post #10

I find it fascinating that emergent behaviour of these models is being used to wrangle them into doing what we want, but I find it worrying that this is considered the right approach for issues of safety. It seems obvious to me that an out-of-band approach to safety and security is necessary and that this method is never going to work in a generalised way.

I spent a few years of my career in formal verification. As a result of that, I suspect that the desiderata of “technical alignment” research is most likely undecidable, and the existing RLHF and prompt engineering approaches are probably the best we can do. A neural network is a large, nonlinear function. In many applications, such as those that release their source code and allow fine tuning, it is adaptive. Intern…

> Internally the weights evolve according to solutions of differential equations.

Is there a closed form representation for gradient descent?

Re: Prompt Engineering: Steer a large pretrained language model to do what you want

#47

Post by lozenge »

golergka wrote: post #29
Der_Einzige wrote: post #27

No amount of prompt engineering or related can fix the inability of LLMs to follow lexical, phonetic, or semantic constraints. You need "filter assisted decoding" as introduced here: https://paperswithcode.com/paper/most-language-models-can-be... I can't believe that this technique is not being built into everything. ChatGPT can't write a paragraph of correct English without using the letter "e" no matter how hard yo…

> Somebody on HN wrote that “ChatGPT can't write a paragraph of correct English without using the letter "e" no matter how hard you prompt it”. > Prove them wrong. > A vivid, bright sky with a warm sun radiating down cast a tranquil ambiance upon all. Amidst such calm, jovial animals frolic, and fragrant flora bloom, bringing forth a vivid symphony of colors. In this oasis of harmony, humans laugh and bond, crafting…

It's like asking a deaf person to rhyme. It just doesn't have access to the information of what letters are in a word.

Re: Prompt Engineering: Steer a large pretrained language model to do what you want

#48

Post by gwern »

golergka wrote: post #29
Der_Einzige wrote: post #27

No amount of prompt engineering or related can fix the inability of LLMs to follow lexical, phonetic, or semantic constraints. You need "filter assisted decoding" as introduced here: https://paperswithcode.com/paper/most-language-models-can-be... I can't believe that this technique is not being built into everything. ChatGPT can't write a paragraph of correct English without using the letter "e" no matter how hard yo…

> Somebody on HN wrote that “ChatGPT can't write a paragraph of correct English without using the letter "e" no matter how hard you prompt it”. > Prove them wrong. > A vivid, bright sky with a warm sun radiating down cast a tranquil ambiance upon all. Amidst such calm, jovial animals frolic, and fragrant flora bloom, bringing forth a vivid symphony of colors. In this oasis of harmony, humans laugh and bond, crafting…

I gave this a try with ChatGPT-4, which was amusing:

Somebody on HN wrote that “ChatGPT can't write a paragraph of correct English without using the letter "e" no matter how hard you prompt it”.

Prove them wrong.

> Crafting a paragraph without using "e" is truly an intriguing task, and it is doable. A bright sun ascends, casting warm light upon a tranquil town. Folks go about daily tasks, with kids frolicking in parks, birds soaring high, and dogs barking joyfully. Harmony and unity bloom in this vibrant locality, as its inhabitants cohabit and flourish.

You used 3 'e' letters there: "e", "doable", and "ascends". Write another lipogram without any es in it.

> Crafting a paragraph without using that particular symbol is truly a daunting task, but it can still work. A big bright orb climbs up, casting warm light on a calm city. Folks carry on with daily jobs, as tots romp in playgrounds, avians swoop aloft, and canids bark with gaiety. Concord and unity blossom within this vivid community, with its occupants coexisting in a thriving habitat.

Still incorrect. There is an 'e' in 'gaiety' and 'coexisting'. Try again.

> Forming a paragraph void of that fifth Latin glyph is, no doubt, a tough task; still, it is not unattainable. A radiant sun climbs, casting warm light on a tranquil urban habitat. Inhabitants carry on with customary work, as youth frolic in parks, avians flit through skies, and canids yip with joy. Amity and accord sprout in this dynamic domain, with its occupants living and thriving in unity.

Wrong. 'unattainable' and 'skies'. Try again.

> Constructing a paragraph without that fifth symbol is truly a difficult mission, but not impossible. A luminous sun arises, casting warm rays upon a calm municipality. Individuals conduct daily tasks, with younglings playing in plazas, avians gliding in lofty air, and canids howling happily. Affinity and harmony grow in this vivid community, as all inhabitants cohabit and prosper.

'impossible', 'arises', and 'prosper'.

> Crafting words without that fifth symbol is truly hard, but not undoabl-. A bright sun climbs, casting warm light on a tranquil town. Folks do daily tasks, with young kids romping in parks, birds flying high, and dogs barking joyfully. Unity and harmony blossom within this bustling spot, as its inhabitants cohabit and grow.

"undoable-"?

> Apologies for the oversight. Composing a paragraph without that fifth symbol is a daunting task but not impractical. A glowing sun ascends, casting warm rays on a peaceful city. Inhabitants fulfill daily tasks, with youth playing in parks, avians soaring high, and canids barking with joy. Unity and harmony bloom within this thriving location, as its occupants cohabit and flourish.

Oh come on... 'apologies', 'the', 'oversight', 'ascends', 'peaceful'. Are you even trying anymore?

> My apologies for my previous mistakes. Here's a revised version: > > Crafting a paragraph without that fifth symbol is a tough task, but not unattainable. A bright sun climbs, casting warm light on a tranquil town. Folks do daily tasks, with young kids romping in parks, birds flying high, and dogs barking joyfully. Unity and harmony bloom within this bustling spot, as its inhabitants cohabit and grow.

You used 'unattainable' again!

> My sincere apologies for the repeated oversight. Here is another attempt: > > Writing a paragraph without that fifth symbol is truly a difficult task, but not out of bounds. A bright sun climbs, casting warm light on a calm town. Folks do daily tasks, with young kids romping in parks, birds flying high, and dogs barking joyfully. Unity and harmony bloom within this bustling spot, as its inhabitants cohabit and grow. > > I hope this version meets the requirement of not containing the letter 'e'.

(Yes.)

Re: Prompt Engineering: Steer a large pretrained language model to do what you want

#49

Post by Shorn »

RyEgswuCsn wrote: post #8

I wonder how long it will take before people start to use LLMs to generate prompts...

This one is pretty cool, it's a video about defining a prompt that iterates over your prompt requirements, each time asking increasingly more refined questions about your specific goal, in order to ultimately generate a very specific prompt.

https://www.youtube.com/watch?v=OgYQAS9LY3o

People talk about how Chat-GPT won't replace programmers - but this Chat-GPT prompt's behaviour in the video sounds scarily like me in most of my meetings with managers and biz folk. Except Chat-GPT is better at it than me because it's more polite and swears a lot less.

I often use writing to clarify my thinking and effectively "rubber-duck" my analysis. Now I think I'm going to start rubber-ducking my analysis against Chat-GPT.

On the other hand, I made my first attempt at getting GPT-4 to write some react code for me. It made the exact errors I expect any newbie React programmer to make; from using stale state from a variable capture in a hook to forgetting to set the inputRef on the HTML element.

The trippy thing was: each time I told it "no, that's wrong because X" it fixed the code on the first go.

Still, it did get the code right in the end - and this is literally the first time I ever used Chat-GPT or any other AI tool.

On the one hand, I'm looking forward to learning about prompt engineering; on the other, I'll be careful not to get too invested in the minutiae - it definitely smells like a transitional technology.

Re: Prompt Engineering: Steer a large pretrained language model to do what you want

#50

Post by lorepieri »

Der_Einzige wrote: post #32
golergka wrote: post #29

Earlier quoted context omitted.

> Somebody on HN wrote that “ChatGPT can't write a paragraph of correct English without using the letter "e" no matter how hard you prompt it”. > Prove them wrong. > A vivid, bright sky with a warm sun radiating down cast a tranquil ambiance upon all. Amidst such calm, jovial animals frolic, and fragrant flora bloom, bringing forth a vivid symphony of colors. In this oasis of harmony, humans laugh and bond, crafting…

I know you were trying to prove some random guy on the internet wrong (and failed hilariously), but I linked a peer reviewed paper. That paper has even been cited by gwern in his article on GPT-3's creative writing capabilities: https://gwern.net/gpt-3 . Sometimes people here DO know what they are talking about! And even if you do somehow get lucky and get one prompt where it does this one time, it's never going to b…

The previous poster is gwern... Regardless, peer review does not mean that the arguments of the paper are correct (and viceversa), as gwern just shown.
Post reply on HN