I’m doing experiments with LLMs and I’m trying to research techniques for grounding. Example prompt templates, for instance. There’s lots of generic articles about grounding, but details and specific examples are thin on the ground. I’ve read the source for langchain to find the prompt template for agent based reasoning, but that was just one perspective…are there better ways?
Ask HN: Where to learn the cutting edge of prompt engineering?
1–10 of 20 posts
Re: Ask HN: Where to learn the cutting edge of prompt engineering?
#2My opinion is if you want to find out what works best is to come up with a bunch of different variations in a context-free environment to not influence prior results, determine some metrics you are targeting, and start prompting away.
Then you will find the answer that works for you, and probably well more thought out than 3/4 of the articles you will find regarding this sort of thing.
Re: Ask HN: Where to learn the cutting edge of prompt engineering?
#3My opinion is if you want to find out what works best is to come up with a bunch of different variations in a context-free environment to not influence prior results, determine some metrics you are targeting, and start prompting away. Then you will find the answer that works for you, and probably well more thought out than 3/4 of the articles you will find regarding this sort of thing.
Prompt Engineering is clearly "a thing" irrespective of whether or not one trains or build models. LLMs clearly have a wide range of possible outputs given a particular prompt (even with just tuning temperature, top_p, top_k) but then, modification of a prompt can lead to significant improvements in the output. it's not a science. It's not really an art either. Certain prompts lead to better outputs than other prompts, and having a systematic way to characterize these differences is going to be important going forward.
I personally stay abreast of new models coming out and run an evals set against new models to assess their performance vs other models (say, gpt-2, gpt-3.5-turbo, etc, gpt-4.)
In terms of grounding, there is RAG, which can be built in any number of ways (PG+pg_vector, vector store, graph db). I would look at arxiv.org publicatons to stay on top of SOTA prompting stuff, as well as adjacent publications (LLMs, scaling, other things)
Re: Ask HN: Where to learn the cutting edge of prompt engineering?
#4Here's a great article that links to a lot of research: https://lilianweng.github.io/posts/2023-03-15-prompt-enginee...
Re: Ask HN: Where to learn the cutting edge of prompt engineering?
#5Here's a great article that links to a lot of research: https://lilianweng.github.io/posts/2023-03-15-prompt-enginee...
> March 15, 2023
Is this obsolete? Does it contain the cutting edge prompt engineering techniques such as saying you'll tip 200$ for a correct answer?
Re: Ask HN: Where to learn the cutting edge of prompt engineering?
#6Is there a "cutting edge"? The space seems pretty pseudo-sciency
I'm reading some papers on arxiv right now, and trying to implement them in our codebase at work. Those papers usually involve doing some common sense thing and measuring the results. Anyone could have come up with it, but they did the data science and showed some evidence it worked.
If there is a better way, I would love to know lol
Re: Ask HN: Where to learn the cutting edge of prompt engineering?
#7Is there a "cutting edge"? The space seems pretty pseudo-sciency I'm reading some papers on arxiv right now, and trying to implement them in our codebase at work. Those papers usually involve doing some common sense thing and measuring the results. Anyone could have come up with it, but they did the data science and showed some evidence it worked. If there is a better way, I would love to know lol
two cents: any situation involving billions/trillions of variables looks pretty pseudo-sciency because you can't reduce it down or isolate components very well. People can do studies, add things and take things out, and sort of hint at things and explain things sort of. It is what it is.
Re: Ask HN: Where to learn the cutting edge of prompt engineering?
#8Is there a "cutting edge"? The space seems pretty pseudo-sciency I'm reading some papers on arxiv right now, and trying to implement them in our codebase at work. Those papers usually involve doing some common sense thing and measuring the results. Anyone could have come up with it, but they did the data science and showed some evidence it worked. If there is a better way, I would love to know lol
two cents: any situation involving billions/trillions of variables looks pretty pseudo-sciency because you can't reduce it down or isolate components very well. People can do studies, add things and take things out, and sort of hint at things and explain things sort of. It is what it is.
Real science is reproducible and provides testable, falsifiable hypotheses.
In practice, some models (ChatGPT in particular) are not deterministic. This makes reproducing things harder. Not impossible, but harder.
Re: Ask HN: Where to learn the cutting edge of prompt engineering?
#9My opinion is if you want to find out what works best is to come up with a bunch of different variations in a context-free environment to not influence prior results, determine some metrics you are targeting, and start prompting away. Then you will find the answer that works for you, and probably well more thought out than 3/4 of the articles you will find regarding this sort of thing.
Prompt Engineering is clearly "a thing" irrespective of whether or not one trains or build models. LLMs clearly have a wide range of possible outputs given a particular prompt (even with just tuning temperature, top_p, top_k) but then, modification of a prompt can lead to significant improvements in the output. it's not a science. It's not really an art either. Certain prompts lead to better outputs than other prompt…
What kind of eval set do you use?
Re: Ask HN: Where to learn the cutting edge of prompt engineering?
#10Earlier quoted context omitted.
two cents: any situation involving billions/trillions of variables looks pretty pseudo-sciency because you can't reduce it down or isolate components very well. People can do studies, add things and take things out, and sort of hint at things and explain things sort of. It is what it is.
Real science is reproducible and provides testable, falsifiable hypotheses. In practice, some models (ChatGPT in particular) are not deterministic. This makes reproducing things harder. Not impossible, but harder.
It's kind of dream, is hard to to deterministically where the path is.