Smart trick, but assumes the “dumb” llm is smart enough not to derail into an article about the lives of South American red ants. Obvious exaggeration, the point being outcomes should stay strictly within topic, avoid unrelated bloat and hit the target.
Don't classify, hallucinate
41–50 of 118 posts
Re: Don't classify, hallucinate
#42[flagged]
Re: Don't classify, hallucinate
#43I would propose the following, query vector store for 10 closest categories based on a query, feed it to an LLM, in the prompt ask it to produce a single digit 0-9 representing the number of the most appropriate choice. Use plain text prompt, dont inflate token count with JSON. There you go, you just drastically reduced the output pricing. Additionally you could experiment with a reranker instead of an LLM or after r…
[deleted]
Re: Don't classify, hallucinate
#44I would propose the following, query vector store for 10 closest categories based on a query, feed it to an LLM, in the prompt ask it to produce a single digit 0-9 representing the number of the most appropriate choice. Use plain text prompt, dont inflate token count with JSON. There you go, you just drastically reduced the output pricing. Additionally you could experiment with a reranker instead of an LLM or after r…
Or press 9 to hear these options again
Your call is important to us. Please listen carefully, as our menu options have changed.
Re: Don't classify, hallucinate
#45Since you map each breadcrumb of the path, how do you deal with differing lengths that would be more appropriate?
Re: Don't classify, hallucinate
#46No? This is just giving up and hoping.
Re: Don't classify, hallucinate
#47Nice trick. Couldn't you embed the query though, compare it to the embedding of the categories, then ship only categories that are close to it in the prompt to a smaller model?
Yeah I had the same question. What's the point of the intermediate step?
Re: Don't classify, hallucinate
#48No? This is just giving up and hoping.
No change from regular chatbot coding, then.
Re: Don't classify, hallucinate
#49Prompt expansion of input to extra categories makes sense if your embedding isn’t working well. But on its own, why use the LLM at all? I think you could have demonstrated the original step first and then shown that it’s useful.
Re: Don't classify, hallucinate
#50I can’t believe programming is now at the stage where advice like "first have the computer give you totally wrong answers, then just find a function that maps the wrong answers to the correct ones!" is a thing.