Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Beating GPT-5.6 Sol on retrieval with 100x cheaper open models

neon.com

Post reply on HN
Print view View original

111–120 of 132 posts

Re: Beating GPT-5.6 Sol on retrieval with 100x cheaper open models

#111

Post by kumama »

nullbio wrote: post #81

I (and I imagine many others) would love to use something like this, but can't, because my data is too sensitive to be uploaded to a cloud of which I have no gaurantees of privacy/security. Is there any way we do this using rented GPUs and open-source software stacks? Paying for the service isn't the issue, I don't care if it's free or if a cut is taken in some capacity, I just don't want the provider to have access…

castform founder here. unfortunately, we are cloud-hosted at this point.

but some easy options on the open-source side include huggingface's trl & unsloth.

you can run our data-generation scripts here: https://github.com/castform-ai/benchmax and then hook it up trl/unsloth for training. should be able to do all of this on your own compute.

Re: Beating GPT-5.6 Sol on retrieval with 100x cheaper open models

#112

Post by softwaredoug »

softwaredoug wrote: post #90

People are building agentic search one of three ways: 1. Actually good retireval. There’s been a lot of progress on serving the kinds of queries agents tend to serve, from places like Hornet, MoxedBread, LightOn. Particularly in late interaction 2. Smarter harnesses with models/judges validating the result. This is now just seen as the generator/ evaluator pattern. Here’s where people try to just use grep or some oth…

*MixedBread

https://www.mixedbread.com/

Re: Beating GPT-5.6 Sol on retrieval with 100x cheaper open models

#113

Post by maxrumpf »

kumama wrote: post #108
softwaredoug wrote: post #90

People are building agentic search one of three ways: 1. Actually good retireval. There’s been a lot of progress on serving the kinds of queries agents tend to serve, from places like Hornet, MoxedBread, LightOn. Particularly in late interaction 2. Smarter harnesses with models/judges validating the result. This is now just seen as the generator/ evaluator pattern. Here’s where people try to just use grep or some oth…

+1 on SID-1. we were definitely inspired by that paper

SID-1 co-author here: was a lot of fun writing it. want to chat? just firstname at domain.

Re: Beating GPT-5.6 Sol on retrieval with 100x cheaper open models

#116

Post by kumama »

punnerud wrote: post #114

What’s in that “Search tool”, could actually be the same solution presented in a different way

castform founder here. it uses lakebases's native bm25 and vector search and fuses the results using rrf (https://medium.com/@devalshah1619/mathematical-intuition-beh...)

Re: Beating GPT-5.6 Sol on retrieval with 100x cheaper open models

#117

Post by softwaredoug »

punnerud wrote: post #114

What’s in that “Search tool”, could actually be the same solution presented in a different way

Usually in these cases you don't need to do much tuning to the retriever. So you just give it BM25 or somesuch.

I'm hesitant to say absolutely zero tuning, because there are cases where you do want to say, bias towards trustworthy results or recent results etc to help the model avoid wasting tokens. But probably not much beyond that.

You can also just create a param in the tool for the agent that selects for "recent" or "popular" or "trustworthy" in ranking.

Re: Beating GPT-5.6 Sol on retrieval with 100x cheaper open models

#118

Post by dwaltrip »

nikcub wrote: post #30
mrinterweb wrote: post #2

There is so much opportunity for purpose built models like this. Ideally a harness should spin up a subagent to offload to targeted models for specific tasks like this. I know this is not a novel idea. Claude code does some of this by handing off the "explore" agent work to haiku. I just love seeing that specialized LLMs are being developed.

There has been an over-obsession with frontier models and benchmarks. Most of the work will be done by task specific models. You don't put Phds on the factory floor.

This only works if the tasks are actually specific and don’t benefit from broad competency.

IMO, this doesn’t match most things that people use LLMs for.

Post reply on HN