[dead]
Beating GPT-5.6 Sol on retrieval with 100x cheaper open models
71–80 of 133 posts
Re: Beating GPT-5.6 Sol on retrieval with 100x cheaper open models
#72Why do we need to train the model to solve for retrieval within the org, so we have to keep training it whenever new dataset is introduced , or am I missing something here ?
Re: Beating GPT-5.6 Sol on retrieval with 100x cheaper open models
#73Why do we need to train the model to solve for retrieval within the org, so we have to keep training it whenever new dataset is introduced , or am I missing something here ?
It's a matter of cost. Did you see the 100x cheaper?
If you have a workload that is going to be very heavy, incurring a large training cost to make a cheaper model work well with the dataset will be dramatic cost reduction. Most large AI workloads can't afford, or truly need, the expense or capability of GPT 5.6 Sol when cheaper models can do.
Of course you could skip that and just use GPT-5.6 Sol everywhere instead. If you're running a fast food restaurant you could hire Michelin star chefs to make your burger and fries without further training. Or you could have a training program for teenagers, a sourcing program, etc. to scale up to your chain to still get consistent quality without needing that level of cost in each store, but replacing it with a centralized repeatable process.
Re: Beating GPT-5.6 Sol on retrieval with 100x cheaper open models
#74Why do we need to train the model to solve for retrieval within the org, so we have to keep training it whenever new dataset is introduced , or am I missing something here ?
(founder of castform here) the model you post-train should ideally learn general patterns & search strategies over your dataset that should transfer to new docs you add to the search corpus (unless its super out of distribution)
Re: Beating GPT-5.6 Sol on retrieval with 100x cheaper open models
#75There is so much opportunity for purpose built models like this. Ideally a harness should spin up a subagent to offload to targeted models for specific tasks like this. I know this is not a novel idea. Claude code does some of this by handing off the "explore" agent work to haiku. I just love seeing that specialized LLMs are being developed.
Yep. It's a real shame that the labs are incentivized not to go in this direction. They all want to try and suck us into the cloud and take away full control and local processing, but there's far more opportunity by building small AI systems and tools into the harness itself to make the models more intelligent.
They naturally don't like this direction, because it draws the intelligence away from their systems, and onto the local machine, where idea moats cannot be protected and hidden, and costs can be dramatically cut. Imagine though, how powerful our harnesses could be if the best researchers were thinking about how to utilize the power of the gaming GPUs that most PC users have (or can get), to supplement the frontier model processing. Instead of trying to have the frontier model do everything, the frontier model can serve as the orchestrator over all of the smaller dedicated harness models. Right now my rtx4090 sits there unused for most of the day while I'm paying for inference in the cloud... It's such a waste of parallel intelligence bandwidth.
I'm not just talking about LLMs either, most people seem unaware that there are a plethora of dedicated AI models for all sorts of conceivable pipeline usecases, from all sorts of classification tasks all the way down to things like code duplication detection. Right now the LLMs completely suck at cleaning up code and architecture, and a big part of that is because the frontier LLM cannot fit your entire codebase + all of its long-chain reasoning into the context window. But using small local models and tools bypasses this problem because small fast models can iterate over an entire codebase quickly. A harness that creates a big model bundle + routing system + DAG-based memory/execution management over all of these has the potential to be incredibly powerful.
Even better, building a framework around this concept and having the frontier model dynamically and adaptively generate the ideal execution system for any given task/domain. We're working on coding today? Okay, here's a recipe we can use: ..., and it generates a local model pipeline execution system that it feeds all of your prompts through in real time by using pre-defined or shared recipe building blocks, etc... Lots of interesting possibilities.
Re: Beating GPT-5.6 Sol on retrieval with 100x cheaper open models
#76There is so much opportunity for purpose built models like this. Ideally a harness should spin up a subagent to offload to targeted models for specific tasks like this. I know this is not a novel idea. Claude code does some of this by handing off the "explore" agent work to haiku. I just love seeing that specialized LLMs are being developed.
> Claude code does some of this by handing off the "explore" agent work to haiku. That is not handing off to a specialized model, its just handing off to a lighter and interior model (compared to the parent model). That by itself can create issues like the lighter model not capturing all the data that the parent needs. The idea is that we get specialized models that are better then general purpose models. But its rar…
> The idea is that we get specialized models that are better then general purpose models. But its rare for a specialized model to beat a strong general model.
Are you sure about that? I mean, MoE is basically an array of specialized models. This beats single-model general purpose performance.
Clever routing systems also seem to indicate frontier or frontier-beating capabilities at reduced cost, such as Fugu.
Further to this though, I think it hasn't been pushed hard enough. We're not going down far enough in the abstraction chain. Most peoples idea of 'specialized models' is an LLM trained on a particular domain (like math vs coding vs creative writing), where-as it should be going lower-level than that and incorporating not only more abstract and smaller specializations of LLMs, but entirely different types of AI models, classifiers, etc.
I also think the reason we don't really hear much about this (yet) is because there's no incentive for the best AI researchers/labs to go down this direction because it's better suited for local-processing over your entire codebase, but the labs all want you to perform inference on their servers and spend your money there, rather than spend your own compute to increase intelligence. They have a larger incentive to try and scale in different ways. We'll have to rely on open-source for this one.
Re: Beating GPT-5.6 Sol on retrieval with 100x cheaper open models
#77I have done my own testing and found that smaller models can beat their larger siblings on fact retrieval from documents. I haven’t investigated it in depth with a large enough dataset but my guess is that larger models overthink it while smaller ones just do it. I would like if they compared this with 5.6 Luna instead.
Anecdotally, it feels like Opus, Fable, and Sol "get distracted" when you use them for writing code. Great at reasoning and coordination but they will go off on a tangent and refactor half the code base. I only use them for reasoning (of course) and coordinating subagents.
mind sharing hints/links on your harness/flow setup?
I did several attempts with naive prompting, but spent more time babysitting than actual flow
Re: Beating GPT-5.6 Sol on retrieval with 100x cheaper open models
#78This feels like the database equivalent of "use the right data structure." We've spent two years assuming the biggest general-purpose model should do everything. It makes more sense for retrieval, reranking, reasoning, and generation to each have their own optimized model if the routing cost is negligible.
Yes, and it's only logical things move in this direction because there's massive hardware incentive to do so. If frontier models can be broken down into small models, networks of smaller-GPUs can be utilized. Right now the smaller GPUs are basically paper weights for frontier intelligence.
Re: Beating GPT-5.6 Sol on retrieval with 100x cheaper open models
#79[dead]
Re: Beating GPT-5.6 Sol on retrieval with 100x cheaper open models
#80I am not sure about GPT-5.6. It usually 10x more verbose for no apparent reason than GPT-5.5. Maybe it is only me.
try adjusting model_verbosity. it defaults to verbose. and of course, use agents.md.