Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Beating GPT-5.6 Sol on retrieval with 100x cheaper open models

neon.com

Post reply on HN
Print view View original

91–100 of 132 posts

Re: Beating GPT-5.6 Sol on retrieval with 100x cheaper open models

#91

Post by typ »

podocarp wrote: post #86
typ wrote: post #63

Earlier quoted context omitted.

Electricity becoming a commodity doesn't necessarily make nuclear reactors become commodities. And as we can see, the publication of open-weight models doesn't really diffuse the know-how of training SoAT models as we expected. Even the major cloud providers are still unable to build competitive ones from scratch on their own. That's very different than the traditional FOSS ecosystem, where everyone can copy, learn,…

I don't think the big companies not making Sota models being an indicator that open source failed. If all semiconductor patents expired tomorrow it would still be prohibitively difficult to manufacture your own CPU at home or at work.

Me neither. But open weights are not open source in the first place. It wouldn't be surprising that the raw intelligence of an LLM model has little to do with the architecture (which is the part most published). The secret sauce, like how/what data gets curated and cleaned, has rarely been talked about. So, even if the labs upload their weights, competitors are very difficult, if not impossible, to replicate the technology independently.

Re: Beating GPT-5.6 Sol on retrieval with 100x cheaper open models

#92

Post by boonzeet »

mrinterweb wrote: post #2

There is so much opportunity for purpose built models like this. Ideally a harness should spin up a subagent to offload to targeted models for specific tasks like this. I know this is not a novel idea. Claude code does some of this by handing off the "explore" agent work to haiku. I just love seeing that specialized LLMs are being developed.

This is how Sakana's Fugu model works, achieving similar performance to Opus/Fable with a mix of open source & mainstream LLMs.

Re: Beating GPT-5.6 Sol on retrieval with 100x cheaper open models

#93

Post by oh_no »

kumama wrote: post #74
linux_devil wrote: post #72

Why do we need to train the model to solve for retrieval within the org, so we have to keep training it whenever new dataset is introduced , or am I missing something here ?

(founder of castform here) the model you post-train should ideally learn general patterns & search strategies over your dataset that should transfer to new docs you add to the search corpus (unless its super out of distribution)

i'm a little confused by this, should most piles of company documentation look pretty similar? what are we getting by tuning at the org level?

what if you have a bunch of teams or apps that have different documentation patterns?

how much does this degrade over time, it beats leading models with that static data set but clearly this edge will degrade with data drift, how quickly does that happen?

also, any "this is 100x cheaper" blogpost means nothing if not discussing TCO (I know your team didn't write this.) I don't care what inference costs are if I don't know training/overhead costs. what's the breakeven point. and again, how long is this RAG stack going to be worth keeping, you beat 5.6 Luna but at some point un-tuned models will beat you, so this is a temporary solution that needs to be re-upped at some point. benchmarks against data drift would help there

Re: Beating GPT-5.6 Sol on retrieval with 100x cheaper open models

#97

Post by kumama »

simianwords wrote: post #83
kumama wrote: post #57

Earlier quoted context omitted.

castform founder here. the roi optimization makes sense. i think there are lots of usecases for which even a 2% gain in accuracy can be quite useful. off the top of my head - high volume customer support. higher accuracy means fewer escalation, reducing labor costs - fraud detection. catching even one extra fraud attempt could mean a lot in savings - and ofc the classic ads use-case where at scale bps in improvement…

Would a bigger model be able to beat yours if some effort were put into prompt?

the bigger model would still cost more :) at the same time, i see prompting as being orthogonal to post-training. i'd imagine post-training a smaller model with a better prompt would make it perform even better

Re: Beating GPT-5.6 Sol on retrieval with 100x cheaper open models

#98

Post by jillesvangurp »

Models without tools and harnesses are not really that useful. My observation is that the tool ux is driving most progress at this point. There are of course open source tools and harnesses but they require more effort to setup properly.

The key challenge is to pick the right model for the right task or sub task and doing that automatically rather than manually. A big part of the problem here is that everybody is picking the most expensive and resource intensive models by default just in case they hit something that is a bit more difficult to get right. It's overkill. Most work people actually do is completely routine and would not have been a challenge for most of the mainstream OSS models.

I'm starting to suffer a bit from model fatigue. There are announcements almost on a daily basis about this or that new model. I can't keep up with that and I don't have time to try them out or evaluate them. I don't want to waste brain cycles on which one to use. I just want to get shit done without micromanaging AI models.

All this marketing BS and confusing naming isn't helping either. It seems a lot of that is just about tricking people into picking the expensive model so they'll burn through more tokens.

Re: Beating GPT-5.6 Sol on retrieval with 100x cheaper open models

#99

Post by childintime »

typ wrote: post #63
cmiles8 wrote: post #42

The big lab models are academically interesting but business wise they seem toast long term. There’s no way for these model companies to compete when the models are becoming a pure commodity and others offering options that are orders of magnitude cheaper. It’s not that the big labs couldn’t theoretically just also put out 100x cheaper options but their business model requires them to generate huge revenues from high…

Electricity becoming a commodity doesn't necessarily make nuclear reactors become commodities. And as we can see, the publication of open-weight models doesn't really diffuse the know-how of training SoAT models as we expected. Even the major cloud providers are still unable to build competitive ones from scratch on their own. That's very different than the traditional FOSS ecosystem, where everyone can copy, learn,…

> Electricity becoming a commodity doesn't necessarily make nuclear reactors become commodities.

Bad comparison? Nuclear reactors become obsolete. The only reason to build a nuclear reactor is as a government vanity project, just to get the bribes you want, or weapons grade material. Dirty tech.

Re: Beating GPT-5.6 Sol on retrieval with 100x cheaper open models

#100

Post by kumama »

kinnth wrote: post #84
mrinterweb wrote: post #2

There is so much opportunity for purpose built models like this. Ideally a harness should spin up a subagent to offload to targeted models for specific tasks like this. I know this is not a novel idea. Claude code does some of this by handing off the "explore" agent work to haiku. I just love seeing that specialized LLMs are being developed.

isn't this also the threat to frontier AI houses? As in they want you to expend tokens in their ecosystem, but the optimization at 100x is their profit?

castform founder here. i'd say it's a threat but the frontier ai labs' argument would basically be that the market opportunity for intelligence is infinite so it doesn't matter. at the same time, i do believe there will continue to be a massive, growing market for big labs, esp for super-frontier use-cases. today that's longer-horizon coding tasks but in the future it can be things like scientific discovery, etc.
Post reply on HN