Ask HN: How do you version GPT prompts

#1

Post by okoma » Sat, Apr 01, 2023, 2:38 PM UTC

This is for the purpose of tracking improvements of a code completion or a chatGPT prompt.

Are you using an existing solution, rolling your own or a mix?

Re: Ask HN: How do you version GPT prompts

#2

Post by verdverm » Sat, Apr 01, 2023, 2:40 PM UTC

Git

Re: Ask HN: How do you version GPT prompts

#3

Post by okoma » Sat, Apr 01, 2023, 4:54 PM UTC

verdverm wrote: post #2

Git

What structure do you use if you wanted to check the accuracy of an older prompt with an updated or extended data set?

Re: Ask HN: How do you version GPT prompts

#4

Post by verdverm » Sat, Apr 01, 2023, 6:00 PM UTC

okoma wrote: post #3

verdverm wrote: post #2

Git

What structure do you use if you wanted to check the accuracy of an older prompt with an updated or extended data set?

This question is slightly different because you have to track a testing set and the metrics. I would look to the many ML workflow tools

Re: Ask HN: How do you version GPT prompts

#5

Post by okoma » Sun, Apr 02, 2023, 9:49 AM UTC

verdverm wrote: post #4

okoma wrote: post #3

Earlier quoted context omitted.

What structure do you use if you wanted to check the accuracy of an older prompt with an updated or extended data set?

This question is slightly different because you have to track a testing set and the metrics. I would look to the many ML workflow tools

Thanks for pointing towards the right direction. I'll edit the original question.

To rephrase, I am looking for a tool to do model lifecycle management https://github.com/kelvins/awesome-mlops#model-lifecycle and wonder if there is any one in particular that you'd think is better suited for prompts, i.e. an array of objects with templated text

Re: Ask HN: How do you version GPT prompts

#6

Post by okoma » Sun, Apr 02, 2023, 11:42 AM UTC

I found a bunch of resources here: https://alphasec.io/prompt-engineering-playgrounds-marketpla...

Good candidates for what I am looking are OpenAI Playground (once it supports versioning and variables for chat), PromptLayer (once it supports chat prompts), EveryPrompt (chat prompts missing). More dev oriented, LangChain https://js.langchain.com/docs/getting-started/guide-chat which can then be easily versioned.