Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Lessons after a Half-billion GPT Tokens

kenkantzer.com

Post reply on HN
Print view View original

21–30 of 179 posts

Re: Lessons after a Half-billion GPT Tokens

#21

Post by gdiamos »

mind-blight wrote: post #4
Yacovlewis wrote: post #2

Interesting piece! My experience around Langchain/RAG differs, so wanted to dig deeper: Putting some logic around handling relevant results helps us produce useful output. Curious what differs on their end.

I suspect the biggest difference is the input data. Embeddings are great over datasets that look like FAQs and QA docs, or data that conceptually fits into very small chunks (tweets, some product reviews, etc). It does very badly over diverse business docs, especially with naive chunking. B2B use cases usually have old PDFs and word docs that need to be searched, and they're often looking for specific keywords (e.g.…

rare words are out of vocab errors in vectors

Especially if they aren’t in the token vocab

Re: Lessons after a Half-billion GPT Tokens

#23

Post by swalsh »

The being too precise reduces accuracy example makes sense to me based on my crude understanding on how these things work.

If you pass in a whole list of states, you're kind of making the vectors for every state light up. If you just say "state" and the text you passed in has an explicit state, than fewer vectors specific to what you're searching for light up. So when it performs the soft max, the correct state is more likely to be selected.

Along the same lines I think his /n vs comma comparison probably comes down to tokenization differences.

Re: Lessons after a Half-billion GPT Tokens

#24

Post by Oras »

eigenvalue wrote: post #12

I agree with most of it, but definitely not the part about Claude3 being “meh.” Claude3 Opus is an amazing model and is extremely good at coding in Python. The ability to handle massive context has made it mostly replace GPT4 for me day to day. Sounds like everyone eventually concludes that Langchain is bloated and useless and creates way more problems than it solves. I don’t get the hype.

+1 for Claude Opus, it had been my go to for the last 3 weeks compared to GPT4. The generated texts are much better than GPT4 when it comes to follow the prompt.

I also tried the API for some financial analysis of large tables, the response time was around 2 minutes, still did it really well and timeout errors were around 1 to 2% only.

Re: Lessons after a Half-billion GPT Tokens

#25

Post by legendofbrando »

The finding on simpler prompts, especially with GPT4 tracks (3.5 requires the opposite).

The take on RAG feels application specific. For our use-case where having details of the past rendered up the ability to generate loose connections is actually a feature. Things like this are what I find excites me most about LLMs, having a way to proxy subjective similarities the way we do when we remember things is one of the benefits of the technology that didn’t really exist before that opens up a new kind of product opportunity.

Re: Lessons after a Half-billion GPT Tokens

#27

Post by ilaksh »

I recently had a bug where I was sometimes sending the literal text "null " right in front of the most important part of my prompt. This caused Claude 3 Sonnet to give the 'ignore' command in cases where it should have used one of the other JSON commands I gave it.

I have an ignore command so that it will wait when the user isn't finished speaking. Which it generally judges okay, unless it has 'null' in there.

The nice thing is that I have found most of the problems with the LLM response were just indications that I hadn't finished debugging my program because I had something missing or weird in the prompt I gave it.

Re: Lessons after a Half-billion GPT Tokens

#28

Post by codewithcheese »

CuriouslyC wrote: post #10

If you used better prompts you could use a less expensive model. "return nothing if you find nothing" is the level 0 version of giving the LLM an out. Give it a softer out ("in the event that you do not have sufficient information to make conclusive statements, you may hypothesize as long as you state clearly that you are doing so, and note the evidence and logical basis for your hypothesis") then ask it to evaluate…

Yeah also prompts should not be developed in abstract. Goal of a prompt is to activate the models internal respentations for it to best achieve the task. Without automated methods, this requires iteratively testing the models reaction to different input and trying to understand how it's interpreting the request and where it's falling down and then patching up those holes.

Need to verify if it even knows what you mean by nothing.

Re: Lessons after a Half-billion GPT Tokens

#29

Post by codewithcheese »

lordofmoria wrote: post #8
sroussey wrote: post #6

Earlier quoted context omitted.

If someone were to create something new, a blank slate approach, what would you find valuable and why?

This is a great question! I think we now know, collectively, a lot more about what’s annoying/hard about building LLM features than we did when LangChain was being furiously developed. And some things we thought would be important and not-easy, turned out to be very easy: like getting GPT to give back well-formed JSON. So I think there’s lots of room. One thing LangChain is doing now that solves something that IS ver…

It can be faster and more effective to fallback to a smaller model (gpt3.5 or haiku), the weakness of the prompt will be more obvious on a smaller model and your iteration time will be faster

Re: Lessons after a Half-billion GPT Tokens

#30

Post by codewithcheese »

_pdp_ wrote: post #17

The biggest realisation for me while making ChatBotKit has been that UX > Model alone. For me, the current state of AI is not about questions and answers. This is dumb. The presentation matters. This is why we are now investing in generative UI.

How are you using Generative UI?
Post reply on HN