A $500 RL fine-tune of a 9B open model beat frontier models on catalog review • Hacker News Board

Re: A $500 RL fine-tune of a 9B open model beat frontier models on catalog review

#111

Post by giancarlostoro » Tue, Jul 28, 2026, 5:19 PM UTC

cmiles8 wrote: post #13

The point that the major labs don’t seem to get is that the vast majority of use cases simply don’t need models that have 50 PhDs and can speak 12 languages. Most use cases are defined within constraints where costs matter a lot. As open weight models and cheap fine tuning services become the norm the whole economic framework of these mega models the labs are in an arms race building just completely crumbles. As does…

I mostly agree, but at the same time I wouldn't blindly ignore the "it's China" angle at all. These are models you cannot open up and do a thorough audit of. I do think local compute will eventually be good enough that most people will just use local models. Microsoft might be the ultimate winner of that if they can stop ruining Windows and focus on making an OS similar to how straightforward and simple macOS is, no ads literally everywhere for Microsoft Office. Just make a good OS Microsoft, is that so much to ask? You HAD a really solid OS and you ruined it.

Re: A $500 RL fine-tune of a 9B open model beat frontier models on catalog review

#112

Post by dimgl » Tue, Jul 28, 2026, 5:20 PM UTC

npn wrote: post #80

fine tuning a small LLMs or even real small language models (like bert) is what the recommended way since the introduction of LLM. the benefit is pretty much obvious: faster to run, fully controlling the stack, better fit for the custom domain... but in practice not many people do the fine tuning, for pretty much a single reason: large language models API cost are still very cheap, fast enough, and get improvement al…

It's also really hard. And you need a lot of data for the fine tuning to produce good results.

Re: A $500 RL fine-tune of a 9B open model beat frontier models on catalog review

#113

Post by synergy20 » Tue, Jul 28, 2026, 5:58 PM UTC

bugglebeetle wrote: post #34

synergy20 wrote: post #31

Earlier quoted context omitted.

what are those cheap fine-tuning services these days?

Tinker is pretty cheap. Prime Intellect if you want more flexibility.

unsloth for small use cases, tinker for enterprise level use cases?

Re: A $500 RL fine-tune of a 9B open model beat frontier models on catalog review

#114

Post by billmalarky » Tue, Jul 28, 2026, 6:18 PM UTC

cmiles8 wrote: post #13

The point that the major labs don’t seem to get is that the vast majority of use cases simply don’t need models that have 50 PhDs and can speak 12 languages. Most use cases are defined within constraints where costs matter a lot. As open weight models and cheap fine tuning services become the norm the whole economic framework of these mega models the labs are in an arms race building just completely crumbles. As does…

>As does the economic picture that justified the massive infrastructure building that’s now broadly funded by a complex network of debt.

I have a genuine question and I'd like to hear people's good faith thoughts on this.

There's a fair case that open models are threatening to institutions who spent a lot on training proprietary SOTA models.

But to my understanding, the massive investment spend (much of it debt backed as you note) is on data centers, chips, physical infra.

Yes, this infra is needed to train the models, but it is also needed to serve inference.

Perhaps the costs associated with training SOTA models is ultimately a "bust" given open models eroding the SOTA closed model performance advantage.

But demand for inference is skyrocketing and there seems to be no end in sight.

The physical hardware underpinning inference is in fact a scarce good (currently, and this seems sustainable at least over mid-term). And inference is a scarce service as such.

I know that cost of inference constantly goes down as models, technical infrastructure, and applied AI techniques become more efficient (specialized SLMs etc). So this puts downward pressure on prices.

But still... demand for inference is just growing like crazy regardless. Putting upwards pressure on prices.

Doesn't this mean that all the spending on AI infra is much better positioned to get positive ROI regardless of the type of model being served?

Put another way, models seem to be commoditizing, but physical hardware is not (currently).

The vast majority of the AI boom spend is on hardware to my understanding (even training capex can be repurposed for inference).

Doesn't this suggest that the economics for the "railroads level of build-out spending" are healthier than they might seem at first glance?

Re: A $500 RL fine-tune of a 9B open model beat frontier models on catalog review

#115

Post by denverllc » Tue, Jul 28, 2026, 6:20 PM UTC

geysersam wrote: post #52

majormajor wrote: post #10

Earlier quoted context omitted.

Sometimes that'll turn up real bugs, sometimes just overengineered designs, premature-optimization, and 1-in-a-million possibility "bugs". And sometimes it's not about the model, it's just about refining the search space. E.g. I've had Opus write tests and GPT 5.5 write the implementation passing all the tests. Then ask about that specific implementation and find some real corner cases. Add those to tests, etc. But t…

They do tend to overengineer. The other day 5.6 Sol generated a while loop around a uuid4 call to make sure the generated ids were unique... I wonder if that's also how I'd behave if I had a reinforcement learning harness around me that dived hard on and punished me for every small mistake.

That is not overengineering, it’s creating job security:

> Funny story no one will believe, but it’s true. A good friend of mine joined a startup as CTO 10 years ago, high growth phase, maybe 200 devs… In his first week he discovered the company had a microservice for generating new UUIDs. One endpoint with its own dedicated team of 3 engineers …including a database guy (the plot thickens). Other teams were instructed to call this service every time they needed a new ‘safe’ UUID. My pal asked wtf. It turned out this service had its own DB to store every previously issued UUID. Requests were handled as follows: it would generate a UUID, then ‘validate’ it by checking its own database to ensure the newly generated UUID didn’t match any previously generated UUIDs, then insert it, then return it to the client. Peace of mind I guess. The team had its own kanban board and sprints.

https://news.ycombinator.com/item?id=48061235

Re: A $500 RL fine-tune of a 9B open model beat frontier models on catalog review

#116

Post by npn » Tue, Jul 28, 2026, 6:59 PM UTC

dimgl wrote: post #112

npn wrote: post #80

fine tuning a small LLMs or even real small language models (like bert) is what the recommended way since the introduction of LLM. the benefit is pretty much obvious: faster to run, fully controlling the stack, better fit for the custom domain... but in practice not many people do the fine tuning, for pretty much a single reason: large language models API cost are still very cheap, fast enough, and get improvement al…

It's also really hard. And you need a lot of data for the fine tuning to produce good results.

well usually you can just generate the data using LLM. use 2 or 3 different frontier models from different providers, then compare the results and pick the consensus. yes it is not 100% correct like when you make it manually, but then again even human makes mistakes.

Re: A $500 RL fine-tune of a 9B open model beat frontier models on catalog review

#117

Post by dominotw » Tue, Jul 28, 2026, 7:10 PM UTC

krapht wrote: post #105

dominotw wrote: post #84

Earlier quoted context omitted.

There is no such thing as extracting "task specific intelligence" into a cheaper model.

News to me. What do you call mixture-of-experts and model distillation?

those are still general purpose models. do you have an example of task specific model?

Re: A $500 RL fine-tune of a 9B open model beat frontier models on catalog review

#118

Post by teravor » Tue, Jul 28, 2026, 7:23 PM UTC

it helps when you first start by distilling (full distribution) domain specific workflows from a larger open model (kimi k3 would be perfect for this now, we used glm 5.2) and then apply your own private RL pipeline to the result. we did this to get a private 27B model for English -> SQL and the results were better than Fable, it lost the ability to explain however.

beating frontier models on highly specialized domains with a tiny fraction of the cost is not difficult to achieve, even when you don't have your own RL pipeline such distillation can still have major savings (27B model will become almost as good as a 3T model on the task). it probably won't work if the task is so ambitious as to not be suited for a specialist.

Re: A $500 RL fine-tune of a 9B open model beat frontier models on catalog review

#119

Post by gen220 » Tue, Jul 28, 2026, 7:33 PM UTC

h_mirin wrote: post #41

Every time I see this kind of story, two things bother me. First, I have watched the free improvement of frontier models surpass the gains from retraining, many times now. Squeezing more out of the models that already exist, or simply doing nothing and waiting, is a real strategy and it often pays better. The fair comparison is not against today's frontier but against whatever ships while you are still maintaining yo…

On your second part:

Many of the most valuable companies in the world (who, incidentally, are spending the most money on frontier model inference), already have "the expensive part", the labeled dataset.

The cost of maintaining the datasets and the models is getting commoditized by startups like braintrust and huggingface.

Re: A $500 RL fine-tune of a 9B open model beat frontier models on catalog review

#120

Post by teravor » Tue, Jul 28, 2026, 7:36 PM UTC

amelius wrote: post #90

Aurornis wrote: post #85

Earlier quoted context omitted.

> The point that the major labs don’t seem to get is that the vast majority of use cases simply don’t need models that have 50 PhDs and can speak 12 languages. Most generic use cases do benefit from a model that has been trained broadly. When you don’t know the specific use case ahead of time, you have to have world knowledge ready to go. Even when coding it’s helpful to have all that knowledge on tap so the model ca…

Can't you take a broadly trained model and throw away useless knowledge (like foreign languages, and 99% of wikipedia) and still end up with a model that is as intelligent and useful as the original but much smaller?

if you have the model weights or a provider that will give you the full token distributions you can have the next best thing and do a domain-specific distillation. this way you can take a 9B or 27B LLM and uplift it to the level of the larger model.

it still takes some work and ingenuity, you need to design a workflow that would let you efficiently probe the larger model with your domain-specific prompts, this usually involves a workflow where you obtain the inverse mapping from frontier models (don't need distributions for this step).