Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Beating GPT-5.6 Sol on retrieval with 100x cheaper open models

neon.com

Post reply on HN
Print view View original

101–110 of 132 posts

Re: Beating GPT-5.6 Sol on retrieval with 100x cheaper open models

#101

Post by drob518 »

jillesvangurp wrote: post #98

Models without tools and harnesses are not really that useful. My observation is that the tool ux is driving most progress at this point. There are of course open source tools and harnesses but they require more effort to setup properly. The key challenge is to pick the right model for the right task or sub task and doing that automatically rather than manually. A big part of the problem here is that everybody is pic…

I agree with this. Interestingly, I’ve been using Deepseek v4 Flash a lot these days but I definitely have to constrain it a lot with tests. Fortunately, I can have it write the tests. It’s extremely cost effective. Still trying to figure out whether the latest update last week that made it smarter actually translates into something I can see in the output. It’s not dumber, but it’s still an open question as to whether it’s “real world smarter.”

Re: Beating GPT-5.6 Sol on retrieval with 100x cheaper open models

#102

Post by richwater »

seahyinghang8 wrote: post #45
richwater wrote: post #8

One thing that plagues [insert current FAANG] is the large amount of corpus knowledge that is outdated/misleading or just plain wrong. I'm curious how this addresses that if it's deriving the reward function from the corpus itself.

(founder of castform here!) - having worked at FAANG / big tech, i totally get this. our example was on gitlab's open source company handbook but i think a real company's corpus is way more messy and has many sources of truth. a few ideas i have yet to validate are: - prioritize recently updated docs when generating the training questions (assumption those docs are more correct than others) - actually including contr…

Thanks! I think those could all be good heuristics.

Re: Beating GPT-5.6 Sol on retrieval with 100x cheaper open models

#103

Post by kumama »

oedemis wrote: post #96

whats about self-consistency like in grpo with majority voiting?

castform founder here. i'm personally a little against techniques like self-consistency/majority voting during rl training because they tend to result in the model's output distribution "sharpening" a lot. this means the model will lose it's exploration ability and probably won't be able to explore/discover new solution strategies, which can be harmful for both rl training + generalization to unseen cases

Re: Beating GPT-5.6 Sol on retrieval with 100x cheaper open models

#104

Post by LunaSea »

childintime wrote: post #99
typ wrote: post #63

Earlier quoted context omitted.

Electricity becoming a commodity doesn't necessarily make nuclear reactors become commodities. And as we can see, the publication of open-weight models doesn't really diffuse the know-how of training SoAT models as we expected. Even the major cloud providers are still unable to build competitive ones from scratch on their own. That's very different than the traditional FOSS ecosystem, where everyone can copy, learn,…

> Electricity becoming a commodity doesn't necessarily make nuclear reactors become commodities. Bad comparison? Nuclear reactors become obsolete. The only reason to build a nuclear reactor is as a government vanity project, just to get the bribes you want, or weapons grade material. Dirty tech.

And you know the small benefit of having stable, large scale and low CO2 emission electricity production.

Re: Beating GPT-5.6 Sol on retrieval with 100x cheaper open models

#105

Post by dd8601fn »

kumama wrote: post #40
try-working wrote: post #28

Earlier quoted context omitted.

yes, and this is why we need model routing

castform founder here: totally! we also think model routing is also a post-training problem i.e. getting a model to predict the difficulty of a task and match it to the right model -> we're gonna be sharing more on that soon :)

I’ve been trying to do this with a pet project and admit that I’m getting terrible results. My small llm as a classifier/router stage just isn’t getting the job done.

Re: Beating GPT-5.6 Sol on retrieval with 100x cheaper open models

#106

Post by drob518 »

mrinterweb wrote: post #2

There is so much opportunity for purpose built models like this. Ideally a harness should spin up a subagent to offload to targeted models for specific tasks like this. I know this is not a novel idea. Claude code does some of this by handing off the "explore" agent work to haiku. I just love seeing that specialized LLMs are being developed.

IMO, purpose built or “adapted” models are The Next Big Thing. If I’m using a model to write Python code, for instance, I really want the 8B or 27B expert model for exactly that, which would also be runnable locally. I don’t care about the 1.8T model that can answer every query under the sun and that only runs in a remote data center.

Re: Beating GPT-5.6 Sol on retrieval with 100x cheaper open models

#107

Post by kumama »

oh_no wrote: post #93
kumama wrote: post #74

Earlier quoted context omitted.

(founder of castform here) the model you post-train should ideally learn general patterns & search strategies over your dataset that should transfer to new docs you add to the search corpus (unless its super out of distribution)

i'm a little confused by this, should most piles of company documentation look pretty similar? what are we getting by tuning at the org level? what if you have a bunch of teams or apps that have different documentation patterns? how much does this degrade over time, it beats leading models with that static data set but clearly this edge will degrade with data drift, how quickly does that happen? also, any "this is 10…

very useful questions here!

- if you have teams that have totally different documentation patterns, yes i think a model trained one teams' docs probably won't directly transfer to another. but let's say you have a model trained on SEC 10-K filings, it should work well as new 10-K filings are added (since terminology & structure of these docs are similar) - on TCO, the training costs for the experiments you saw were <$200 - you bring up a good point on data drift, don't have concrete numbers/experiments here. something we should defn look into (though i imagine this also changes a lot based on use-case and company)

Re: Beating GPT-5.6 Sol on retrieval with 100x cheaper open models

#108

Post by kumama »

softwaredoug wrote: post #90

People are building agentic search one of three ways: 1. Actually good retireval. There’s been a lot of progress on serving the kinds of queries agents tend to serve, from places like Hornet, MoxedBread, LightOn. Particularly in late interaction 2. Smarter harnesses with models/judges validating the result. This is now just seen as the generator/ evaluator pattern. Here’s where people try to just use grep or some oth…

+1 on SID-1. we were definitely inspired by that paper

Re: Beating GPT-5.6 Sol on retrieval with 100x cheaper open models

#110

Post by kumama »

srvraw wrote: post #85
kumama wrote: post #74

Earlier quoted context omitted.

(founder of castform here) the model you post-train should ideally learn general patterns & search strategies over your dataset that should transfer to new docs you add to the search corpus (unless its super out of distribution)

could you share more about what you mean by "general patterns & search strategies"? I can think of it being along the lines of searching over specific tables or databases for queries in certain context. It's an exciting line of work and I'm interested because I need something like this for the problem I'm solving atm. So, I'd like to understand how the training generalizes

yup! it’s mostly about getting better at using the right search keywords.

for more complex multi-hop question, it's also about knowing which sections of a document to look up and in what order.

Post reply on HN