turns out the bottleneck was never model size, it was having someone who actually understood the problem define the reward function
A $500 RL fine-tune of a 9B open model beat frontier models on catalog review
51–60 of 141 posts
Re: A $500 RL fine-tune of a 9B open model beat frontier models on catalog review
#52Earlier quoted context omitted.
The upper end is all about coding. If I have terra on extra high write code, Sol will find a plethora of bugs and rip the code apart. Anything else? Sure use a cheaper model.
Sometimes that'll turn up real bugs, sometimes just overengineered designs, premature-optimization, and 1-in-a-million possibility "bugs". And sometimes it's not about the model, it's just about refining the search space. E.g. I've had Opus write tests and GPT 5.5 write the implementation passing all the tests. Then ask about that specific implementation and find some real corner cases. Add those to tests, etc. But t…
They do tend to overengineer. The other day 5.6 Sol generated a while loop around a uuid4 call to make sure the generated ids were unique...
I wonder if that's also how I'd behave if I had a reinforcement learning harness around me that dived hard on and punished me for every small mistake.
Re: A $500 RL fine-tune of a 9B open model beat frontier models on catalog review
#53The point that the major labs don’t seem to get is that the vast majority of use cases simply don’t need models that have 50 PhDs and can speak 12 languages. Most use cases are defined within constraints where costs matter a lot. As open weight models and cheap fine tuning services become the norm the whole economic framework of these mega models the labs are in an arms race building just completely crumbles. As does…
But then can't the case be made that narrower and stricter-defined use cases are better served by more conventional ML? If/Wherever efficiency is a concern, that is.
Re: A $500 RL fine-tune of a 9B open model beat frontier models on catalog review
#54Every time I see this kind of story, two things bother me. First, I have watched the free improvement of frontier models surpass the gains from retraining, many times now. Squeezing more out of the models that already exist, or simply doing nothing and waiting, is a real strategy and it often pays better. The fair comparison is not against today's frontier but against whatever ships while you are still maintaining yo…
Regarding your first point, if the cost of not doing it while you wait exceeds the cost of training/maintaining the model, then this approach still makes sense. This really depends on how fast you expect cheap (and open-source) models to improve. So it might only be a temporary strategy, but still worthwhile. Also, I expect that specialized models will always be better (cheaper or better outcomes) than a general model, similar to how specialized HW like GPUs is still used even though CPUs have improved a lot too.
Maintaining in the way of ongoing training is relatively cheap, considering that the initial training run is only 500$ (in this example). Creating new examples to account for drift in the training data is more expensive, but these examples can then be reused when training a new model. And to some degree you need them anyways, to evaluate the models and prompt changes you make.
Overall, as you pointed out, this won't always make sense (either due to the cost of creating examples and training, or simply the lack of available data). But they point this out themselves in the diagram towards the bottom of the page: it only makes sense for frequent and verifiable tasks. This might result in this only being a sensible approach for very large companies (they talk about millions of decisions), but it might make sense for them.
Re: A $500 RL fine-tune of a 9B open model beat frontier models on catalog review
#55Earlier quoted context omitted.
Data drift. The data you are doing real world inference on can start changing, meaning your model performance starts degrading, so the model needs to be retrained on new data (that you have to collect and prepare).
Sure, but that apparently only costs $500. If you do that once a month then that's still basically free. Hell if you do it once a week it's still about as much as a single cheap employee.
you missed the parent comment's point that the $500 training run was merely the last step, generating the training data is likely to be significantly more expensive and time-consuming. might still pay for itself, but it's not a trivial "spend $500 every month" decision.
Re: A $500 RL fine-tune of a 9B open model beat frontier models on catalog review
#56The point that the major labs don’t seem to get is that the vast majority of use cases simply don’t need models that have 50 PhDs and can speak 12 languages. Most use cases are defined within constraints where costs matter a lot. As open weight models and cheap fine tuning services become the norm the whole economic framework of these mega models the labs are in an arms race building just completely crumbles. As does…
But then can't the case be made that narrower and stricter-defined use cases are better served by more conventional ML? If/Wherever efficiency is a concern, that is.
That's assuming conventional ML and fine-tuned SLMs are interchangeable with acceptable behavioral change/degradation for any specific case
Re: A $500 RL fine-tune of a 9B open model beat frontier models on catalog review
#57The point that the major labs don’t seem to get is that the vast majority of use cases simply don’t need models that have 50 PhDs and can speak 12 languages. Most use cases are defined within constraints where costs matter a lot. As open weight models and cheap fine tuning services become the norm the whole economic framework of these mega models the labs are in an arms race building just completely crumbles. As does…
Yeah and these latest and greatest models universally suck hard for any use case outside of the few 'blessed' ones. Like I'm sure their ability to write prose and generally sound like a human being has regressed quite a bit, but even if not. Opus 5 is barely above GPT4 when it comes to stuff like home improvement advice.
Re: A $500 RL fine-tune of a 9B open model beat frontier models on catalog review
#58A fine-tune will generally outperform just about any other method on a closed domain, non generative problem.
LLMs are great because they can handle open domain problems in part because they are generative.
Re: A $500 RL fine-tune of a 9B open model beat frontier models on catalog review
#59What I really started to notice is that SOTA models are really good at putting themselves out of the job. We can see this already with GPT how luna can do 90% of what sol is used for. The only reason why china still bothers 'distilling' models is accurate training data generation, something that oai and anthropic had to spend years collecting while trying to dodge legal challenges. The more intelligent models get, th…
This is a general theme with technology and the 'S-curve'. Let's not even get into whether the improvement for AI reasoning ability has slowed - for practical purposes of writing a React frontend, it has.
But other tech is like that - I don't even remember when I bought my LCD TV - 2018 I think? I have no inclination of buying a new one.
Technology has a tendency to replace new technology, or intrude into vacant areas, but its very rare for technology to replace non-technology (like human interaction).
Most of the recreation humans do in front of screens is tending to (para)social relationships.
Re: A $500 RL fine-tune of a 9B open model beat frontier models on catalog review
#60Earlier quoted context omitted.
Sometimes that'll turn up real bugs, sometimes just overengineered designs, premature-optimization, and 1-in-a-million possibility "bugs". And sometimes it's not about the model, it's just about refining the search space. E.g. I've had Opus write tests and GPT 5.5 write the implementation passing all the tests. Then ask about that specific implementation and find some real corner cases. Add those to tests, etc. But t…
> even the lower-end models can find the things they didn't think of first when inspecting the already-written output I use Deepseek V4 Pro for my hobby project (an OpenCode client). The economics will obviously be different at work. It’s worse than the GPTs, but indeed, if you focus it on reviewing its own code (essentially spending more reasoning and changing perspective), it’s also quite capable at improving its o…
On my own time I still like the 'chat' format of me writing a short prompt and having the model turn it into a few dozen lines of code, or make a simple refactor (like turn a field into a method parameter.
I wasn't impressed with Chinese models (I tried Deepseek V4 Pro/Flash, and GLM5.2) the amount of questionable code and mistakes and misunderstandings meant the money saved and faster speed of models didn't translate to faster progress for me.