2026 will be the year of specialized SLMs...enterprises care about more IP ownership/control, lower costs, and higher quality than the slow and expensive generic models that were not optimized for their use cases.
The case for the return of fine-tuning
71–80 of 85 posts
Re: The case for the return of fine-tuning
#72I ran a survey on Twitter over the past few days asking for successful case studies that produced economically valuable results from fine-tuning LLMs. I ask a version of this every six months or so, and usually the results are quite disappointing. This time I had more credible replies than I have had in the past. Here's my thread with highlights: https://twitter.com/simonw/status/1979254349235925084 And in a thread v…
I imagine it's pretty bad risk to reward ratio for most companies. Especially when just tossing some stuff into your system prompt is an option.
Re: The case for the return of fine-tuning
#73Earlier quoted context omitted.
There is also a reason why you don’t have general purpose applications. Most users understand that Excel is for data tables and Paint is for images even though some people have fun playing with the boundary and creating Excel paintings.
This is exactly the intuition that leads to excitement about fine tuning. However, I personally think that this intuition applies to products and interfaces, not to AI. Intelligence and learning is general. Intelligence without generalization is memorization, which seems to be less useful in practice.
[deleted]
Re: The case for the return of fine-tuning
#74I ran a survey on Twitter over the past few days asking for successful case studies that produced economically valuable results from fine-tuning LLMs. I ask a version of this every six months or so, and usually the results are quite disappointing. This time I had more credible replies than I have had in the past. Here's my thread with highlights: https://twitter.com/simonw/status/1979254349235925084 And in a thread v…
I imagine it's pretty bad risk to reward ratio for most companies. Especially when just tossing some stuff into your system prompt is an option.
Yeah, that's my assumption too. Fine-tuning is really expensive in terms of skills and time needed to attempt it, and there's a very real chance that your attempts will fail to make a meaningful improvement over being smarter with your prompts.
Even worse, even if you DO get an improvement you are likely to find that it was a waste of time in a month or two when then next upgraded version of the underlying models are released.
The places it makes sense from what I can tell are mainly when you are running so many prompts that the cost saving by running a smaller, cheaper model can outweigh the labor and infrastructure costs involved in getting it to work. If your token spend isn't tens (probably hundreds) of thousands of dollars you're unlikely to save money like this.
If it's not about cost saving, the other reasons are latency and being able to achieve something that the root model just couldn't do.
Datadog reported a latency improvement, because fine-tuning let them run a much smaller (and hence faster) model. That's a credible reason if you are building high value features that a human being is waiting on, like live-typing features.
The most likely cases I've heard of for getting the model to do something it just couldn't do before mainly involve vision LLMs, which makes sense to me - training a model to be able to classify images that weren't in the training set might make more sense than stuffing more example images into the prompt (though models like Gemini will accept dozens of not hundreds of comparable images in the prompt, which can then benefit from prompt caching).
The last category is actually teaching it a new skill. The best example here are low-resource programming languages - Jane Street and OCaml or Morgan Stanley and Q for example.
Jane Street OCaml: https://www.youtube.com/watch?v=0ML7ZLMdcl4
Morgan Stanley Q: https://huggingface.co/morganstanley/qqWen-1.5B-SFT
Re: The case for the return of fine-tuning
#75Earlier quoted context omitted.
> Fine-tuning is a good technique to have in a toolbox, but in reality, it is feasible only in some use cases. Yes, 100s of housands of them
Care to elaborate what are some of those use cases?
Almost everything, isn't it?
From fine-tuning for coding assist, to medical applications, customer support, legal and financial use cases, various classification tasks, for government work, statistics, language learning, music, education, even for role playing game character AI...
I'd rather have a fine-tuned model specialized to any of those tasks and countless others when I'm doing one of those tasks, than a jack of all trades...
Re: The case for the return of fine-tuning
#76Earlier quoted context omitted.
Transfer learning is a thing. But the issue with the gap is that the datasets for "applying X" aren't easy to come by.
There is an awful lot of "looking for my keys under the street light" going around these days. I've seen a bunch of projects proposed that are either based on existing data (but have no useful application of that data) or have a specific application (but lack the data and evaluation required to perform that task). It doesn't matter how good your data is if no one has any use for things like it, and it doesn't matter…
The importance of having good metrics cannot be overstated.
On the "applying X" problem - this almost feels to me like another argument against fine tuning? Because it seems like Applying can be a surprisingly broad skill, and frontier lab AIs are getting good at Applying in a broad fashion.
Re: The case for the return of fine-tuning
#77Earlier quoted context omitted.
I imagine it's pretty bad risk to reward ratio for most companies. Especially when just tossing some stuff into your system prompt is an option.
Yeah, that's my assumption too. Fine-tuning is really expensive in terms of skills and time needed to attempt it, and there's a very real chance that your attempts will fail to make a meaningful improvement over being smarter with your prompts. Even worse, even if you DO get an improvement you are likely to find that it was a waste of time in a month or two when then next upgraded version of the underlying models are…
Have you heard of any attempts to bake MCP definitions into LoRA adapters? I've been wondering if that's a viable approach, so you don't have to put them all in context, and toggling them on and off would just be a matter of applying or unapplying the weights. That seems like it'd be more robust than putting "enable FooMCP" "disable FooMCP" etc in the context, which I'd think would trip up the LLM eventually. And it would avoid full rebuild of the KV cache that'd be required if you fully removed FooMCP from the context prefix.
Depending on use case you could either insert the LoRA weights as their own layers at runtime (no time to create, but extra layer to compute each token), merge them with existing layers (initial delay to merge layers, but no runtime penalty after), or have pre-merged models for common cases (no perf penalty but have to reserve more storage).
Re: The case for the return of fine-tuning
#78Just coming out of founding one of the first LLM fine tuning startups - Lamini - I disagree Our thesis was that fine tuning would be easier than deep learning for users to adopt because it was starting from a very capable base LLM rather than starting from scratch However, our main finding with over 20 deployments was that LLM fine tuning is no easier to use than deep learning The current market situation is that ML…
Will Anthropic/OpenAI really hire anyone who can fine-tune an LLM?
I think you misunderstand what they are saying - doing a good job of fine tuning is difficult.
Training an LLM from scratch is trivial - training a good one is difficult. Fine tuning is trivial - doing a good job is difficult. Hitting a golf ball is trivial - hitting a 300 yard drive down the middle of the fairway is difficult.
Re: The case for the return of fine-tuning
#79Earlier quoted context omitted.
Yeah, that's my assumption too. Fine-tuning is really expensive in terms of skills and time needed to attempt it, and there's a very real chance that your attempts will fail to make a meaningful improvement over being smarter with your prompts. Even worse, even if you DO get an improvement you are likely to find that it was a waste of time in a month or two when then next upgraded version of the underlying models are…
Have you heard of any attempts to bake MCP definitions into LoRA adapters? I've been wondering if that's a viable approach, so you don't have to put them all in context, and toggling them on and off would just be a matter of applying or unapplying the weights. That seems like it'd be more robust than putting "enable FooMCP" "disable FooMCP" etc in the context, which I'd think would trip up the LLM eventually. And it…
I've not heard of anyone trying that, but I don't think I've been looking in the right kinds of places.
My current mental model of LoRA is that this would be unlikely to Work, but I've never used them so I don't really know what I'm talking about. Would be a very interesting experiment!
Re: The case for the return of fine-tuning
#80Earlier quoted context omitted.
Users often found it hard to know exactly where the boundaries are. This is a reason why general purpose models shine. You don’t have to carefully characterize a task and put guard rails around it.
There is also a reason why you don’t have general purpose applications. Most users understand that Excel is for data tables and Paint is for images even though some people have fun playing with the boundary and creating Excel paintings.
Interesting you bring up Excel. ChatGPT's chat interface is going to be Excel for the AI era. Everyone knows there's a better interface to be had, but it just works.