81–90 of 133 posts
Is there any way we do this using rented GPUs and open-source software stacks? Paying for the service isn't the issue, I don't care if it's free or if a cut is taken in some capacity, I just don't want the provider to have access to my data.
I (and I imagine many others) would love to use something like this, but can't, because my data is too sensitive to be uploaded to a cloud of which I have no gaurantees of privacy/security. Is there any way we do this using rented GPUs and open-source software stacks? Paying for the service isn't the issue, I don't care if it's free or if a cut is taken in some capacity, I just don't want the provider to have access…
Other than that there is not really a difference to renting a GPU since the GPU provider can also just steal your data.
Local GPU(s) are always an option if you have the possibility. It is also not that difficult to run with stuff like “LocalAI”
Earlier quoted context omitted.
> There is a reason why we hear less about this idea of smaller expert models, because large strong models to the tasks just as good. Smaller models are cheaper, sometimes faster. I agree that the “we’re an LLM fine-tuned for X” hasn’t worked out because you can just train Claude to do X (and Anthropic will), but not burning Opus/Fable tokens on dumb-but-token-heavy tasks is good sense. As we move from “integrate AI…
castform founder here. the roi optimization makes sense. i think there are lots of usecases for which even a 2% gain in accuracy can be quite useful. off the top of my head - high volume customer support. higher accuracy means fewer escalation, reducing labor costs - fraud detection. catching even one extra fraud attempt could mean a lot in savings - and ofc the classic ads use-case where at scale bps in improvement…
There is so much opportunity for purpose built models like this. Ideally a harness should spin up a subagent to offload to targeted models for specific tasks like this. I know this is not a novel idea. Claude code does some of this by handing off the "explore" agent work to haiku. I just love seeing that specialized LLMs are being developed.
Why do we need to train the model to solve for retrieval within the org, so we have to keep training it whenever new dataset is introduced , or am I missing something here ?
(founder of castform here) the model you post-train should ideally learn general patterns & search strategies over your dataset that should transfer to new docs you add to the search corpus (unless its super out of distribution)
The big lab models are academically interesting but business wise they seem toast long term. There’s no way for these model companies to compete when the models are becoming a pure commodity and others offering options that are orders of magnitude cheaper. It’s not that the big labs couldn’t theoretically just also put out 100x cheaper options but their business model requires them to generate huge revenues from high…
Electricity becoming a commodity doesn't necessarily make nuclear reactors become commodities. And as we can see, the publication of open-weight models doesn't really diffuse the know-how of training SoAT models as we expected. Even the major cloud providers are still unable to build competitive ones from scratch on their own. That's very different than the traditional FOSS ecosystem, where everyone can copy, learn,…
Earlier quoted context omitted.
castform founder here. despite our bet on fine-tuned smaller open-source models, i'm still quite bullish on the big labs. i think scaled closed models will continue to dominate for more general purpose use-case like codegen, search, etc. but intelligence has lots of long-tail applications and i think for these longer-tail applications, finetuned custom models will rule
Product search model: https://www.linkedin.com/posts/introducing-ontology-1-ugcPos... Edit: more direct links, sorry: https://onton.com/research/ontology-1 https://onton.com/research/ontology-1-benchmarks
Earlier quoted context omitted.
castform founder here. the roi optimization makes sense. i think there are lots of usecases for which even a 2% gain in accuracy can be quite useful. off the top of my head - high volume customer support. higher accuracy means fewer escalation, reducing labor costs - fraud detection. catching even one extra fraud attempt could mean a lot in savings - and ofc the classic ads use-case where at scale bps in improvement…
Would a bigger model be able to beat yours if some effort were put into prompt?
There is so much opportunity for purpose built models like this. Ideally a harness should spin up a subagent to offload to targeted models for specific tasks like this. I know this is not a novel idea. Claude code does some of this by handing off the "explore" agent work to haiku. I just love seeing that specialized LLMs are being developed.
There has been an over-obsession with frontier models and benchmarks. Most of the work will be done by task specific models. You don't put Phds on the factory floor.
1. Actually good retireval. There’s been a lot of progress on serving the kinds of queries agents tend to serve, from places like Hornet, MoxedBread, LightOn. Particularly in late interaction
2. Smarter harnesses with models/judges validating the result. This is now just seen as the generator/ evaluator pattern. Here’s where people try to just use grep or some other naive retrieval system. Let the agent figure it out. But it’ll consume a lot of tokens to get good results as it iterates and loops.
3. A model trained for retrieval. Give it dumb retriever like in (2) but it is fine tuned on the task as in (1).
This article is 3. But we’ve been seeing this all year with SID.ai, Gleans Waldo model etc. if this interests you I’d check those out, particularly SID.
I wrote about these 3 approaches here https://softwaredoug.com/blog/2026/06/08/three-kinds-of-agen...