Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

GPT-Prompt-Engineer

github.com

Post reply on HN
Print view View original

21–30 of 166 posts

Re: GPT-Prompt-Engineer

#21

Post by vivekv »

Kichererbsen wrote: post #5
PUSH_AX wrote: post #4

It would be cool if, given a handful of test cases, you could send those off to the LLM to generate even more test cases. My first thought when looking over this tool was "Why do I have to do all the work?", the ideal scenario is that I give the high level description and the LLM does the hard work to create the best prompt.

I haven't read the article at all, but what you said just reminded me that I've been using ChatGPT to create Dungeons & Dragons campaigns / adventures / worlds (it's crazy good at that) and one trick i've started doing at the end of a fruitful conversation is to ask it to summarize the discussion so far _in a format to be used as a prompt_. it works more or less.

So one job that has been 'taken over by ai' before it even started ?

Re: GPT-Prompt-Engineer

#22

Post by fluxinflex »

danielbln wrote: post #7
Towaway69 wrote: post #2

Is this prompt generation for the purposes of prompt engineering? Is this then a kind of meta engineering? Engineering for the purposes of engineering which then hopefully will generate working code for the computer that generated the prompt and the response to the prompt.

A bit like code generation, really. Transpile one code to another and have the execution engine run that.

In sense yes, previously we were engaged in "google engineering, then we went to "stackoverflow engineering" and now its "prompt engineering" - with every step, the magic and mystic increases.

Re: GPT-Prompt-Engineer

#23

Post by msp26 »

Currently working on something similar for myself, this doesn't seem to fit my needs (benchmarking generations too rather than just classification). I only have a crude cosine similarity metric for accuracy for now. Also I'm using function calling rather than the normal completions.

I was hoping this would do something more interesting with multiple messages (if using a chat model) rather than just dumping the entire prompt in one message. The assistant lets you do stuff with examples.

Re: GPT-Prompt-Engineer

#24

Post by iRomain »

msp26 wrote: post #23

Currently working on something similar for myself, this doesn't seem to fit my needs (benchmarking generations too rather than just classification). I only have a crude cosine similarity metric for accuracy for now. Also I'm using function calling rather than the normal completions. I was hoping this would do something more interesting with multiple messages (if using a chat model) rather than just dumping the entire…

Cool for you. What about if you contribute to this one or open-source yours?

Re: GPT-Prompt-Engineer

#26

Post by msp26 »

iRomain wrote: post #24
msp26 wrote: post #23

Currently working on something similar for myself, this doesn't seem to fit my needs (benchmarking generations too rather than just classification). I only have a crude cosine similarity metric for accuracy for now. Also I'm using function calling rather than the normal completions. I was hoping this would do something more interesting with multiple messages (if using a chat model) rather than just dumping the entire…

Cool for you. What about if you contribute to this one or open-source yours?

It's company code, I don't think I can open-source it. Happy to answer any questions/discuss prompt benchmarking though.

Re: GPT-Prompt-Engineer

#27

Post by A-Train »

arbol wrote: post #9
Kiro wrote: post #6

How are they actually ranked?

It seems like a `ranking_system_prompt` is used to rank the output of other prompts, which is pretty cool! > Your job is to rank the quality of two outputs generated by different prompts. The prompts are used to generate a response for a given task. You will be provided with the task description, the test prompt, and two generations - one for each system prompt. Rank the generations in order of quality. If Generation…

I'm pretty surprised more people dont use logit biases to call openai with. Checking if something is either a or b means that the tokens for those letters must be 100 weight which means they will be chosen no matter what and no other character is allowed.

Re: GPT-Prompt-Engineer

#28

Post by sgt101 »

this is supervised machine learning on top of unsupervised machine learning with some interesting wrinkles in both steps!

it reminds me of those aircraft that folks in rural india build from time to time.

Re: GPT-Prompt-Engineer

#29

Post by f1shy »

muzani wrote: post #15
mangecoeur wrote: post #11

Should we really call it "engineering" if its a case of "try random things until one of them works without really knowing why"?

Looks more like science here. Run a bunch of experiments and see which does better. But literally the first sentence of the readme is "Prompt engineering is kind of like alchemy."

This comes to mind: http://lambda-the-ultimate.org/node/5335

Re: GPT-Prompt-Engineer

#30

Post by quickthrower2 »

boyka wrote: post #14
mnky9800n wrote: post #13

Earlier quoted context omitted.

That's called machine learning.

Gradient descent type optimization is far from "trying random things until one of them works without really knowing why". You can calculate all partial derivatives and understand the impact.

I think it is a freaking miracle it even works. I understand how it works for say 100 parameter linear regression, but that it would work for billions of parameters (billion dimensional space) by nudging the parameters each a little (based on purely it's impact on the loss, assuming everything stays the same), is not obvious to me. It is a kind of magic.

Regarding randomness, the initialization of the weights is random, and if they use dropouts that is random too, plus the order in which to process the text might be random.

Post reply on HN