Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The case for the return of fine-tuning

welovesota.com

Post reply on HN
Print view View original

21–30 of 85 posts

Re: The case for the return of fine-tuning

#21

Post by qrios »

> Finally, companies may have reached the ceiling of what can be achieved with prompting alone. Some want models that know their vocabulary, their tone, their taxonomy, and their compliance rules.

Together with speed and const, this is from my point of view this is the only "case" for the return of fine-tuning here. And this can be managed by context management.

With growing context sizes, first RAG replaced fine-tuning and later even RAG was replaced by just a good-enough prompt preparation for more and more usage pattern.

Sure, speed and costs are important drivers. But like with FPGAs vs. CPUs or GPUs, the development costs and delivery time for high-performance solutions, eliminate the benefit most the time.

Re: The case for the return of fine-tuning

#22

Post by gdiamos »

Just coming out of founding one of the first LLM fine tuning startups - Lamini - I disagree

Our thesis was that fine tuning would be easier than deep learning for users to adopt because it was starting from a very capable base LLM rather than starting from scratch

However, our main finding with over 20 deployments was that LLM fine tuning is no easier to use than deep learning

The current market situation is that ML engineers who are good enough at deep learning to master fine tuning can found their own AI startup or join Anthropic/OpenAI. They are underpaid building LLM solutions. Expert teams building Claude, GPT, and Qwen will out compete most users who try fine tuning on their own.

RAG, prompt engineering, inference time compute, agents, memory, and SLMs are much easier to use and go very far for most new solutions

Re: The case for the return of fine-tuning

#24

Post by bjornsing »

gdiamos wrote: post #22

Just coming out of founding one of the first LLM fine tuning startups - Lamini - I disagree Our thesis was that fine tuning would be easier than deep learning for users to adopt because it was starting from a very capable base LLM rather than starting from scratch However, our main finding with over 20 deployments was that LLM fine tuning is no easier to use than deep learning The current market situation is that ML…

Will Anthropic/OpenAI really hire anyone who can fine-tune an LLM?

Re: The case for the return of fine-tuning

#25

Post by gdiamos »

bjornsing wrote: post #24
gdiamos wrote: post #22

Just coming out of founding one of the first LLM fine tuning startups - Lamini - I disagree Our thesis was that fine tuning would be easier than deep learning for users to adopt because it was starting from a very capable base LLM rather than starting from scratch However, our main finding with over 20 deployments was that LLM fine tuning is no easier to use than deep learning The current market situation is that ML…

Will Anthropic/OpenAI really hire anyone who can fine-tune an LLM?

They will hire anyone who can produce a model better than GPT5, which is the bar for fine tuning

Otherwise, you should just use gpt5

Preparing a few thousands training examples and pressing fine tune can improve the base LLM in a few situations, but it also can make the LLM worse at other tasks in hard to understand ways that only show up in production because you didn’t build evals that are good enough to catch them. It also has all of the failure modes of deep learning. There is a reason why deep learning training never took off like LLMs did despite many attempts at building startups around it.

Andrej karpathy has a rant about it that captures some of the failure modes of fine tuning - https://karpathy.github.io/2019/04/25/recipe/

Re: The case for the return of fine-tuning

#26

Post by yunwal »

gdiamos wrote: post #25
bjornsing wrote: post #24

Earlier quoted context omitted.

Will Anthropic/OpenAI really hire anyone who can fine-tune an LLM?

They will hire anyone who can produce a model better than GPT5, which is the bar for fine tuning Otherwise, you should just use gpt5 Preparing a few thousands training examples and pressing fine tune can improve the base LLM in a few situations, but it also can make the LLM worse at other tasks in hard to understand ways that only show up in production because you didn’t build evals that are good enough to catch them…

It’s quite easy to produce a model that’s better than GPT-5 at arbitrarily small tasks. As of right now, GPT-5 can’t classify a dog by breed based on good photos for all but the most common breeds, which is like an AI-101 project.

Re: The case for the return of fine-tuning

#27

Post by gdiamos »

yunwal wrote: post #26
gdiamos wrote: post #25

Earlier quoted context omitted.

They will hire anyone who can produce a model better than GPT5, which is the bar for fine tuning Otherwise, you should just use gpt5 Preparing a few thousands training examples and pressing fine tune can improve the base LLM in a few situations, but it also can make the LLM worse at other tasks in hard to understand ways that only show up in production because you didn’t build evals that are good enough to catch them…

It’s quite easy to produce a model that’s better than GPT-5 at arbitrarily small tasks. As of right now, GPT-5 can’t classify a dog by breed based on good photos for all but the most common breeds, which is like an AI-101 project.

Try doing a head to head comparison using all LLM tricks available including prompt engineering, rag, reasoning, inference time compute, multiple agents, tools, etc

Then try the same thing using fine tuning. See which one wins. In ML class we have labeled datasets with breeds of dogs hand labeled by experts like Andrej, in real life users don’t have specific, clearly defined, and high quality labeled data like that.

I’d be interested to be proven wrong

I think it is easy for strong ML teams to fall into this trap because they themselves can get fine tuning to work well. Trying to scale it to a broader market is where it fell apart for us.

This is not to say that no one can do it. There were users who produced good models. The problem we had was where to consistently find these users who were willing to pay for infrastructure.

I’m glad we tried it, but I personally think it is beating a dead horse/llama to try it today

Re: The case for the return of fine-tuning

#28

Post by echelon »

gdiamos wrote: post #22

Just coming out of founding one of the first LLM fine tuning startups - Lamini - I disagree Our thesis was that fine tuning would be easier than deep learning for users to adopt because it was starting from a very capable base LLM rather than starting from scratch However, our main finding with over 20 deployments was that LLM fine tuning is no easier to use than deep learning The current market situation is that ML…

What models did you try to find tune? Were the models at the time even good enough to fine tune? Did they suffer from catastrophic forgetting?

We have a lot of more capable open source models now. And my guess is that if you designed models specifically for being fine tuned, they could escape many of the last generation pitfalls.

Companies would love to own their own models instead of renting from a company that seeks to replace them.

Re: The case for the return of fine-tuning

#29

Post by echelon »

gdiamos wrote: post #27
yunwal wrote: post #26

Earlier quoted context omitted.

It’s quite easy to produce a model that’s better than GPT-5 at arbitrarily small tasks. As of right now, GPT-5 can’t classify a dog by breed based on good photos for all but the most common breeds, which is like an AI-101 project.

Try doing a head to head comparison using all LLM tricks available including prompt engineering, rag, reasoning, inference time compute, multiple agents, tools, etc Then try the same thing using fine tuning. See which one wins. In ML class we have labeled datasets with breeds of dogs hand labeled by experts like Andrej, in real life users don’t have specific, clearly defined, and high quality labeled data like that.…

If your customers can't fine tune, do it for them instead.
Post reply on HN