No mention of training/test/holdout dataset splits anywhere in this article - so how am I to know whether or not this is some overfit result?
A $500 RL fine-tune of a 9B open model beat frontier models on catalog review
101–110 of 141 posts
Re: A $500 RL fine-tune of a 9B open model beat frontier models on catalog review
#102is this an ad?
| Harvey's legal agent beats GPT-5.5 and Claude Opus 4.8 on its own rubrics, and Intercom's Fin Apex resolves more support issues at lower cost.
The rubric and cost argument here just casually ignores all of the other challenges and real business issues of evolving models over time
Re: A $500 RL fine-tune of a 9B open model beat frontier models on catalog review
#103The point that the major labs don’t seem to get is that the vast majority of use cases simply don’t need models that have 50 PhDs and can speak 12 languages. Most use cases are defined within constraints where costs matter a lot. As open weight models and cheap fine tuning services become the norm the whole economic framework of these mega models the labs are in an arms race building just completely crumbles. As does…
Going by the chart in the article, if your total workload is 1k cataloged items and your quality threshold is 70%, why wouldn't you just pay $19 to gemini API instead of $500 + time to make a custom fine tune?
The resultant model much cheaper per call and outperforming frontier models. So you'd quickly make your money back at any sort of meaningful scale, continue to pay less from that point on, and have better results. And I think removing a third party dependency is also a benefit in and of itself.
Re: A $500 RL fine-tune of a 9B open model beat frontier models on catalog review
#104I have great interest in fine-tuning open models, and I'm looking for resources that HN folks can personally recommend. This article looks good and I've bookmarked it to read more thoroughly over time. I've gotten as far as running Nemotron-3-Nano 30b locally, and plan to target models around 30b - 120b parameters. Based on brief examination of the results I can get, I think these vanilla models are capable enough to…
What kind of hardware are you using and what is your token generation speed (tok/s)? Every time I've tried to run local models, even on my beefy mac book (128Gb Ram), I've been very disappointed at tok/s speed.
I've been using dual NVIDIA GPU's with Ollama. Even when I push to 30b models and more context window, Ollama manages RAM / VRAM very well. I never got OOM errors, just massive slowdown as the PCI-E bottleneck throttles the GPU's. I've had some large prompts take 30 minutes to process.
But I should mention: I am trying to implement workflows and processes akin to CI/CD that run 24/7 in the background. These are not interactive use cases, so I don't care so much about tokens per second.
Re: A $500 RL fine-tune of a 9B open model beat frontier models on catalog review
#105The point that the major labs don’t seem to get is that the vast majority of use cases simply don’t need models that have 50 PhDs and can speak 12 languages. Most use cases are defined within constraints where costs matter a lot. As open weight models and cheap fine tuning services become the norm the whole economic framework of these mega models the labs are in an arms race building just completely crumbles. As does…
There is no such thing as extracting "task specific intelligence" into a cheaper model.
News to me. What do you call mixture-of-experts and model distillation?
Re: A $500 RL fine-tune of a 9B open model beat frontier models on catalog review
#106Well I surely hope so. That’s the whole point of fine tuning.
Re: A $500 RL fine-tune of a 9B open model beat frontier models on catalog review
#107The point that the major labs don’t seem to get is that the vast majority of use cases simply don’t need models that have 50 PhDs and can speak 12 languages. Most use cases are defined within constraints where costs matter a lot. As open weight models and cheap fine tuning services become the norm the whole economic framework of these mega models the labs are in an arms race building just completely crumbles. As does…
I’m not sure it will go this direction. Using a generic (near) frontier model is often cheap enough that you need to talk really big volumes before it pays off to fine tune.
My example: we were doing single digit millions of automated call summaries a few years back at a major bank with GPT-4o. Smaller model gave more rejected summaries (compliance not happy), so we briefly looked at fine tuning a smaller model and basically concluded that even at that scale the effort of data collection, management, fine tuning, hosting the model, etc didn’t have a sufficient business case vs picking up other projects.
Re: A $500 RL fine-tune of a 9B open model beat frontier models on catalog review
#108The problem i've had with finetuning models is that most of the time better prompting beats finetuning
Better prompting doesn't improve response time or price!
Ok but a task that works fine on qwen 397b can be finetuned on qwen9b. But in every case so far when building the eval for evaluating the traces I’ve discovered a better prompt that closes the gap better than the finetuning.
Re: A $500 RL fine-tune of a 9B open model beat frontier models on catalog review
#109I didn't read the Ramp article but this reads like a post hoc fallacy. Companies with 2x revenue have money to spend on AI. Companies with 1.15x revenue don't.
Alternatively, AI-heavy companies are much more likely to be more agile growth-stage companies rather than large mature companies.
Re: A $500 RL fine-tune of a 9B open model beat frontier models on catalog review
#110Earlier quoted context omitted.
Can't you take a broadly trained model and throw away useless knowledge (like foreign languages, and 99% of wikipedia) and still end up with a model that is as intelligent and useful as the original but much smaller?
I'm not an AI researcher (would love to hear an actually informed answer on this!), but AFAIK the weights are more or less a black box - you can't easily pinpoint that weights x, y and z relate to Ancient Rome and can therefore be safely removed from a model that's optimised for coding.
You can, and there are many techniques to do so - it's not super precise but you can essentially "delete" concepts out of a model.