Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The case for the return of fine-tuning

welovesota.com

Post reply on HN
Print view View original

31–40 of 85 posts

Re: The case for the return of fine-tuning

#31

Post by yunwal »

gdiamos wrote: post #27
yunwal wrote: post #26

Earlier quoted context omitted.

It’s quite easy to produce a model that’s better than GPT-5 at arbitrarily small tasks. As of right now, GPT-5 can’t classify a dog by breed based on good photos for all but the most common breeds, which is like an AI-101 project.

Try doing a head to head comparison using all LLM tricks available including prompt engineering, rag, reasoning, inference time compute, multiple agents, tools, etc Then try the same thing using fine tuning. See which one wins. In ML class we have labeled datasets with breeds of dogs hand labeled by experts like Andrej, in real life users don’t have specific, clearly defined, and high quality labeled data like that.…

I mean, at the point where you’re writing tools to assist it, we are no longer comparing the performance of 2 LLMs. You’re taking a solution that requires a small amount of expertise, and replacing it with another solution that requires more expertise, and costs more. The question is not “can fine tuning alone do better than every other trick in the book plus a SOTA LLM plus infinite time and money?” The question is: “is fine tuning useful?”

Re: The case for the return of fine-tuning

#32

Post by gdiamos »

echelon wrote: post #29
gdiamos wrote: post #27

Earlier quoted context omitted.

Try doing a head to head comparison using all LLM tricks available including prompt engineering, rag, reasoning, inference time compute, multiple agents, tools, etc Then try the same thing using fine tuning. See which one wins. In ML class we have labeled datasets with breeds of dogs hand labeled by experts like Andrej, in real life users don’t have specific, clearly defined, and high quality labeled data like that.…

If your customers can't fine tune, do it for them instead.

How can you hire enough people to scale that while making the economics work?

Why would they join you rather than founding their own company?

Re: The case for the return of fine-tuning

#33

Post by AYBABTME »

leblancfg wrote: post #13

Fine tuning was never really hard to do locally if you had the hardware. What I’d like to read in an article like this is more details into why they’re making a comeback. Curious to hear others’ thoughts on this

Which minimum hardware spec would qualify as making this not really hard to do locally?

Re: The case for the return of fine-tuning

#35

Post by echelon »

gdiamos wrote: post #32
echelon wrote: post #29

Earlier quoted context omitted.

If your customers can't fine tune, do it for them instead.

How can you hire enough people to scale that while making the economics work? Why would they join you rather than founding their own company?

> How can you hire enough people to scale that while making the economics work?

Pick the right customers.

> Why would they join you rather than founding their own company?

The network effects of having enough resources in one place. For having other teams deal with the training data, infrastructure, deployment, etc.

Re: The case for the return of fine-tuning

#36

Post by gdiamos »

echelon wrote: post #28
gdiamos wrote: post #22

Just coming out of founding one of the first LLM fine tuning startups - Lamini - I disagree Our thesis was that fine tuning would be easier than deep learning for users to adopt because it was starting from a very capable base LLM rather than starting from scratch However, our main finding with over 20 deployments was that LLM fine tuning is no easier to use than deep learning The current market situation is that ML…

What models did you try to find tune? Were the models at the time even good enough to fine tune? Did they suffer from catastrophic forgetting? We have a lot of more capable open source models now. And my guess is that if you designed models specifically for being fine tuned, they could escape many of the last generation pitfalls. Companies would love to own their own models instead of renting from a company that seek…

We used the best models available and went from the Pythia/gpt2 to Deepseek generations.

One annoying part was switching to new and better models that came out literally every week.

I don’t think it substantially changes anything. If anything I think the release of more advanced models like qwen-next makes things like fp4, moe, and reasoning tokens an even higher barrier of entry.

Re: The case for the return of fine-tuning

#37

Post by gdiamos »

yunwal wrote: post #31
gdiamos wrote: post #27

Earlier quoted context omitted.

Try doing a head to head comparison using all LLM tricks available including prompt engineering, rag, reasoning, inference time compute, multiple agents, tools, etc Then try the same thing using fine tuning. See which one wins. In ML class we have labeled datasets with breeds of dogs hand labeled by experts like Andrej, in real life users don’t have specific, clearly defined, and high quality labeled data like that.…

I mean, at the point where you’re writing tools to assist it, we are no longer comparing the performance of 2 LLMs. You’re taking a solution that requires a small amount of expertise, and replacing it with another solution that requires more expertise, and costs more. The question is not “can fine tuning alone do better than every other trick in the book plus a SOTA LLM plus infinite time and money?” The question is:…

Fair didn’t seem to matter to users who just wanted to build solutions with reasonable time and budget

Re: The case for the return of fine-tuning

#38

Post by gdiamos »

gdiamos wrote: post #32
echelon wrote: post #29

Earlier quoted context omitted.

If your customers can't fine tune, do it for them instead.

How can you hire enough people to scale that while making the economics work? Why would they join you rather than founding their own company?

I think you are saying to go after the very high end of the market.

That’s fair, one market segment of this is sometimes called sovereign compute.

Another common model that I have seen is to become the deepmind for one very large and important customer.

I think this works.

Post reply on HN