Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Beating GPT-5.6 Sol on retrieval with 100x cheaper open models

neon.com

Post reply on HN
Print view View original

51–60 of 134 posts

Re: Beating GPT-5.6 Sol on retrieval with 100x cheaper open models

#51

Post by kumama »

Foobar8568 wrote: post #21
aliljet wrote: post #7

There is a more serious question in here that's not being answered. How effective is the retrieval in finding buried needles in larger and larger haystacks. And there's a correlary question, how effective could you be in finding paired needles in that haystack where you need to hold a needle to unlock finding another needle.

Considering the state of the field ( RAG/retrieval/evaluation) I have 0 trust in it, even more if it's closed source with bullshit claim like that. Everything is vibe sloped to death, and dead after a few months to a couple of years (and not hard to be 100 cheaper than GPT-5.6 sol ... DS is basically free and I guess already 100 times cheaper or more, and here another slope ).

castform founder here. we should have made it more prominent on the blog but here's the full code example: https://github.com/castform-ai/benchmax/tree/main/examples/n...

Re: Beating GPT-5.6 Sol on retrieval with 100x cheaper open models

#52

Post by seahyinghang8 »

breadislove wrote: post #15

On what do you guys test the model. Its very dubious that there is no common retrieval benchmark such as browsecomp plus or similar tested. And what metric do you report?

(founder of castform here) - we didn't get to dive too deep into the dataset we were using for the retrieval in the blogpost for brevity, but we did link the training run (which shows the dataset) here: https://app.castform.com/train/a7a898f6-d802-4908-b044-acb81...

the page shows the exact trace of all the models we are comparing against and the aggregate scores

we generated the question & answer pair from gitlab product handbook (https://handbook.gitlab.com/) since the point is to show that you can generate training questions from raw data corpus (something a company already has today)

Re: Beating GPT-5.6 Sol on retrieval with 100x cheaper open models

#54

Post by seahyinghang8 »

JCharante wrote: post #10

I have done my own testing and found that smaller models can beat their larger siblings on fact retrieval from documents. I haven’t investigated it in depth with a large enough dataset but my guess is that larger models overthink it while smaller ones just do it. I would like if they compared this with 5.6 Luna instead.

we actually have the test benchmark against luna too! it's just not in our title but you can see it in the first diagram below the title. luna does pretty well tbh but sol is just a tad bit better. but luna is way cheaper.

if you want to dive down into the various traces of the benchmark, you can check this out: https://app.castform.com/train/a7a898f6-d802-4908-b044-acb81...

- founder of castform

Re: Beating GPT-5.6 Sol on retrieval with 100x cheaper open models

#55

Post by seahyinghang8 »

JCharante wrote: post #10

I have done my own testing and found that smaller models can beat their larger siblings on fact retrieval from documents. I haven’t investigated it in depth with a large enough dataset but my guess is that larger models overthink it while smaller ones just do it. I would like if they compared this with 5.6 Luna instead.

founder of castform here again - slightly unrelated to retrieval but on the topic that folks are discussing here, i was actually collecting benchmarking various coding traces for the purpose of training a model router and surprisingly, luna held up very well against sol and terra. it was able to solve close to >95% the that sol can handle at a fraction of the cost. have not benchmarked the OSS models yet but will add the popular ones to the list like Deepseek Flash and Kimi k3 to see how they fare.

will share the full results soon!

Re: Beating GPT-5.6 Sol on retrieval with 100x cheaper open models

#56

Post by seahyinghang8 »

andai wrote: post #38

Nice, but there's no mention of how Luna or DSFlash perform on the same task? (Being 25x and 50x cheaper respectively.) Nor of how much faster their custom model performs?

we actually have the test benchmark against luna but no deepseek flash (we haven't added DSFlash into our benchmarking model pipeline)

you can check out the full comparison against all the other models here: https://app.castform.com/train/a7a898f6-d802-4908-b044-acb81...

- founder of castform

Re: Beating GPT-5.6 Sol on retrieval with 100x cheaper open models

#57

Post by kumama »

tyre wrote: post #36
benjiro29 wrote: post #22

Earlier quoted context omitted.

> Claude code does some of this by handing off the "explore" agent work to haiku. That is not handing off to a specialized model, its just handing off to a lighter and interior model (compared to the parent model). That by itself can create issues like the lighter model not capturing all the data that the parent needs. The idea is that we get specialized models that are better then general purpose models. But its rar…

> There is a reason why we hear less about this idea of smaller expert models, because large strong models to the tasks just as good. Smaller models are cheaper, sometimes faster. I agree that the “we’re an LLM fine-tuned for X” hasn’t worked out because you can just train Claude to do X (and Anthropic will), but not burning Opus/Fable tokens on dumb-but-token-heavy tasks is good sense. As we move from “integrate AI…

castform founder here. the roi optimization makes sense. i think there are lots of usecases for which even a 2% gain in accuracy can be quite useful. off the top of my head

- high volume customer support. higher accuracy means fewer escalation, reducing labor costs - fraud detection. catching even one extra fraud attempt could mean a lot in savings - and ofc the classic ads use-case where at scale bps in improvement could mean millions in revenue :)

Re: Beating GPT-5.6 Sol on retrieval with 100x cheaper open models

#58

Post by seahyinghang8 »

aliljet wrote: post #7

There is a more serious question in here that's not being answered. How effective is the retrieval in finding buried needles in larger and larger haystacks. And there's a correlary question, how effective could you be in finding paired needles in that haystack where you need to hold a needle to unlock finding another needle.

tldr: we generated synthetic training questions from the gitlab product handbook.

totally agree that this larger corpus with harder to search information would be a good way to stress test - i'm sure we will encounter more interesting problems to solve. love to hear any suggestions of corpus to search against that is not just the public internet

the training run link is also a little buried but here, you can see the comparison against the various models and their exact traces: https://app.castform.com/train/a7a898f6-d802-4908-b044-acb81...

Re: Beating GPT-5.6 Sol on retrieval with 100x cheaper open models

#59

Post by seahyinghang8 »

BedVibe_Studios wrote: post #20

This feels like the database equivalent of "use the right data structure." We've spent two years assuming the biggest general-purpose model should do everything. It makes more sense for retrieval, reranking, reasoning, and generation to each have their own optimized model if the routing cost is negligible.

founder of castform here, we believe that as agent deployment moves from experimentation phase where cost doesn't matter as much to deployment (what's the margin of serving the request), there will be a rise in interest in optimized models.

Re: Beating GPT-5.6 Sol on retrieval with 100x cheaper open models

#60

Post by esafak »

cmiles8 wrote: post #42

The big lab models are academically interesting but business wise they seem toast long term. There’s no way for these model companies to compete when the models are becoming a pure commodity and others offering options that are orders of magnitude cheaper. It’s not that the big labs couldn’t theoretically just also put out 100x cheaper options but their business model requires them to generate huge revenues from high…

Small models aren't going to take people's jobs. Agents using large models might. That's how their numbers make sense.
Post reply on HN