I am not sure about GPT-5.6. It usually 10x more verbose for no apparent reason than GPT-5.5. Maybe it is only me.
Have you tried Claude? 5.6 feels less verbose, and less messy to me.
Beating GPT-5.6 Sol on retrieval with 100x cheaper open models
31–40 of 132 posts
Re: Beating GPT-5.6 Sol on retrieval with 100x cheaper open models
#32Re: Beating GPT-5.6 Sol on retrieval with 100x cheaper open models
#33I have done my own testing and found that smaller models can beat their larger siblings on fact retrieval from documents. I haven’t investigated it in depth with a large enough dataset but my guess is that larger models overthink it while smaller ones just do it. I would like if they compared this with 5.6 Luna instead.
Just an anecdote but thats why Deepseek v4 flash 0731 is my current favorite model. It's really not very "eager" and stays on the task at hand.
Re: Beating GPT-5.6 Sol on retrieval with 100x cheaper open models
#34I am not sure about GPT-5.6. It usually 10x more verbose for no apparent reason than GPT-5.5. Maybe it is only me.
it is definitely more verbose.
Re: Beating GPT-5.6 Sol on retrieval with 100x cheaper open models
#35On what do you guys test the model. Its very dubious that there is no common retrieval benchmark such as browsecomp plus or similar tested. And what metric do you report?
Keeping track of any AI progress is becoming harder by the day, because there's ambiguity around common/clear/consistent benchmarks. Everything is constantly skewed into favourable directions.
TBF gaming benchmarks is not something new to AI
Re: Beating GPT-5.6 Sol on retrieval with 100x cheaper open models
#36There is so much opportunity for purpose built models like this. Ideally a harness should spin up a subagent to offload to targeted models for specific tasks like this. I know this is not a novel idea. Claude code does some of this by handing off the "explore" agent work to haiku. I just love seeing that specialized LLMs are being developed.
> Claude code does some of this by handing off the "explore" agent work to haiku. That is not handing off to a specialized model, its just handing off to a lighter and interior model (compared to the parent model). That by itself can create issues like the lighter model not capturing all the data that the parent needs. The idea is that we get specialized models that are better then general purpose models. But its rar…
> There is a reason why we hear less about this idea of smaller expert models, because large strong models to the tasks just as good.
Smaller models are cheaper, sometimes faster. I agree that the “we’re an LLM fine-tuned for X” hasn’t worked out because you can just train Claude to do X (and Anthropic will), but not burning Opus/Fable tokens on dumb-but-token-heavy tasks is good sense.
As we move from “integrate AI into Y” to “optimize the ROI on Y”, we’ll see more of this.
Re: Beating GPT-5.6 Sol on retrieval with 100x cheaper open models
#37Maybe, but a specific example showing how to do it would have been a more compelling argument.
Re: Beating GPT-5.6 Sol on retrieval with 100x cheaper open models
#38Nice, but there's no mention of how Luna or DSFlash perform on the same task? (Being 25x and 50x cheaper respectively.)
Nor of how much faster their custom model performs?
Re: Beating GPT-5.6 Sol on retrieval with 100x cheaper open models
#39Nice, but there's no mention of how Luna or DSFlash perform on the same task? (Being 25x and 50x cheaper respectively.) Nor of how much faster their custom model performs?
[deleted]
Re: Beating GPT-5.6 Sol on retrieval with 100x cheaper open models
#40There is so much opportunity for purpose built models like this. Ideally a harness should spin up a subagent to offload to targeted models for specific tasks like this. I know this is not a novel idea. Claude code does some of this by handing off the "explore" agent work to haiku. I just love seeing that specialized LLMs are being developed.
yes, and this is why we need model routing
castform founder here: totally! we also think model routing is also a post-training problem i.e. getting a model to predict the difficulty of a task and match it to the right model -> we're gonna be sharing more on that soon :)