31–40 of 56 posts
As an aside: one thing I've tried to use ChatGPT for is to select applicable options from a list. When I index the list as 1..., 2... Etc. I find that the LLM likes to just start printing out ascending numbers.
What I've found kind of works is indexing by African names, e.g Thandokazi, Ntokozo, etc. then the AI seems to have less bias.
Curios what others have done in this case
Agree with lots of this. As an aside: one thing I've tried to use ChatGPT for is to select applicable options from a list. When I index the list as 1..., 2... Etc. I find that the LLM likes to just start printing out ascending numbers. What I've found kind of works is indexing by African names, e.g Thandokazi, Ntokozo, etc. then the AI seems to have less bias. Curios what others have done in this case
Earlier quoted context omitted.
They mention few-shot prompting in the prompt engineering section, which I think is what you mean.
Oh yeah. I read few-shot like it means trying a few times to get an appropriate output. That’s how the author uses the word “shot” in the beginning of the article. Priming is a specific term that means giving examples in the context window. But yeah, the author seems to describe this. Still, you can go a long way with priming. I wouldn’t even think of fine-tuning before trying priming for a good while. It might still…
And the reason we don't really do it is context length. Our contexts are long and complex and there are so many subtleties that I'm worried about either saturating the context window or just not covering enough ground to matter.
Earlier quoted context omitted.
Yep, it's a DAG, though that only occurred to me after we built this so we didn't model it that way at first. It can be the same LLM with different prompts or totally different models, I think there's no rule and it depends on what you're doing + what your benchmarks tell you. We're running it in prod btw, though don't have any code to share.
funnily enough i have a library i’m planning to open source soon! i’ve used airflow as a guideline for it as well.
Earlier quoted context omitted.
Oh yeah. I read few-shot like it means trying a few times to get an appropriate output. That’s how the author uses the word “shot” in the beginning of the article. Priming is a specific term that means giving examples in the context window. But yeah, the author seems to describe this. Still, you can go a long way with priming. I wouldn’t even think of fine-tuning before trying priming for a good while. It might still…
Ha good point, I did say "let's have another shot" when I just meant another try at generating! FWIW "few shot prompting" is how most people refer to this technique, I think (e.g. see https://www.promptingguide.ai/techniques/fewshot ), I haven't heard "priming" before, though it does convey the right thing. And the reason we don't really do it is context length. Our contexts are long and complex and there are so many…
https://github.com/thmsmlr/instructor_ex
It piggybacks on Ecto schemas and works really well (if instructed correctly).
Agree with lots of this. As an aside: one thing I've tried to use ChatGPT for is to select applicable options from a list. When I index the list as 1..., 2... Etc. I find that the LLM likes to just start printing out ascending numbers. What I've found kind of works is indexing by African names, e.g Thandokazi, Ntokozo, etc. then the AI seems to have less bias. Curios what others have done in this case
I'm a little surprised to hear this, my experience has been a little better. Are you using GPT4? I know 3.5 is significantly more challenged/challenging with things like this. It's still possible to make it do the right thing, but much more careful prompting is required.
This is a great write up! I nodded my head thru the whole post. Very much aligns with our experience over the past year. I wrote a simple example (overkiLLM) on getting reliable output from many unreliable outputs here[0]. This doesn't employ agents, just an approach I was interested in trying. I choose writing an H1 as the task, but a similar approach would work for writing any short blob of text. The script generat…
Oh this is fun! So you basically define personalities by picking well-known people that are probably represented in the training data and ask them (their LLM-imagined doppelganger) to vote?
Another approach is to use multiple agents to generate a distribution over predictions, in sort of like bayesian estimation.
My experience with AI agents is that they don't understand nuance. Thie makes sense since they are trained on a wide range of data produced by the masses. The masses aren't good with nuance. That's why, if you put 10 experts together, they will often make worse decisions than they would have made individually. Im terms of coding, I managed to get AI to build a simple working collaborative app but beyond a certain poi…
You are correct that we should be using expert AIs rather than general purpose ones when possible though.