Lessons after a Half-billion GPT Tokens
kenkantzer.com
Lessons after a Half-billion GPT Tokens
1–10 of 179 posts
Re: Lessons after a Half-billion GPT Tokens
#2Interesting piece!
My experience around Langchain/RAG differs, so wanted to dig deeper: Putting some logic around handling relevant results helps us produce useful output. Curious what differs on their end.
Re: Lessons after a Half-billion GPT Tokens
#3For a few uni/personal projects I noticed the same about Langchain: it's good at helping you use up tokens. The other use case, quickly switching between models, is a very valid reason still. However, I've recently started playing with OpenRouter which seems to abstract the model nicely.
Re: Lessons after a Half-billion GPT Tokens
#4Interesting piece! My experience around Langchain/RAG differs, so wanted to dig deeper: Putting some logic around handling relevant results helps us produce useful output. Curious what differs on their end.
I suspect the biggest difference is the input data. Embeddings are great over datasets that look like FAQs and QA docs, or data that conceptually fits into very small chunks (tweets, some product reviews, etc).
It does very badly over diverse business docs, especially with naive chunking. B2B use cases usually have old PDFs and word docs that need to be searched, and they're often looking for specific keywords (e.g. a person's name, a product, an id, etc). Vectors terms to do badly in those kinds of searches, and just returning chunks misses a lot of important details
Re: Lessons after a Half-billion GPT Tokens
#5> This worked sometimes (I’d estimate >98% of the time), but failed enough that we had to dig deeper.
> While we were investigating, we noticed that another field, name, was consistently returning the full name of the state…the correct state – even though we hadn’t explicitly asked it to do that.
> So we switched to a simple string search on the name to find the state, and it’s been working beautifully ever since.
So, using ChatGPT helped uncover the correct schema, right?
Re: Lessons after a Half-billion GPT Tokens
#6For a few uni/personal projects I noticed the same about Langchain: it's good at helping you use up tokens. The other use case, quickly switching between models, is a very valid reason still. However, I've recently started playing with OpenRouter which seems to abstract the model nicely.
If someone were to create something new, a blank slate approach, what would you find valuable and why?
Re: Lessons after a Half-billion GPT Tokens
#7> We consistently found that not enumerating an exact list or instructions in the prompt produced better results
Not sure if he means training here or using his product. I think the latter.
My end-user exp of GPT3.5 is that I need to be - not just precise but the exact flavor of precise. It's usually after some trial and error. Then more error. Then more trial.
Getting a useful result on the 1st or 3rd try happens maybe 1 in 10 sessions. A bit more common is having 3.5 include what I clearly asked it not to. It often complies eventually.
Re: Lessons after a Half-billion GPT Tokens
#8For a few uni/personal projects I noticed the same about Langchain: it's good at helping you use up tokens. The other use case, quickly switching between models, is a very valid reason still. However, I've recently started playing with OpenRouter which seems to abstract the model nicely.
If someone were to create something new, a blank slate approach, what would you find valuable and why?
This is a great question!
I think we now know, collectively, a lot more about what’s annoying/hard about building LLM features than we did when LangChain was being furiously developed.
And some things we thought would be important and not-easy, turned out to be very easy: like getting GPT to give back well-formed JSON.
So I think there’s lots of room.
One thing LangChain is doing now that solves something that IS very hard/annoying is testing. I spent 30 minutes yesterday re-running a slow prompt because 1 in 5 runs would produce weird output. Each tweak to the prompt, I had to run at least 10 times to be reasonably sure it was an improvement.
Re: Lessons after a Half-billion GPT Tokens
#9I feel like for just extracting data into JSON, smaller LLMs could probably do fine, especially with constrained generation and training on extraction.
Re: Lessons after a Half-billion GPT Tokens
#10If you used better prompts you could use a less expensive model.
"return nothing if you find nothing" is the level 0 version of giving the LLM an out. Give it a softer out ("in the event that you do not have sufficient information to make conclusive statements, you may hypothesize as long as you state clearly that you are doing so, and note the evidence and logical basis for your hypothesis") then ask it to evaluate its own response at the end.