Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Fine-tuning LLMs is a waste of time

codinginterviewsmadesimple.substack.com

Post reply on HN
Print view View original

21–30 of 94 posts

Re: Fine-tuning LLMs is a waste of time

#21

Post by rybosome »

I think the point the author misses is that many applications of fine-tuning are to get a model to do a single task. This is what I have done in my current role at my company.

We’ve fine-tuned open weight models for knowledge-injection, among other things, and get a model that’s better than OpenAI models at exactly one hyper specific task for our use case, which is hardware verification. Or, fine-tuned the OAI models and get significantly better OAI models at this task, and then only use them for this task.

The point is that a network of hyper-specific fine-tuned models is how a lot of stuff is implemented. So I disagree from direct experience with the premise that fine-tuning is a waste of time because it is destructive.

I don’t care if I “damage” Llama so that it can’t write poetry, give me advice on cooking, or translate to German. In this instance I’m only ever going to prompt it with: “Does this design implement the AXA protocol? ”

Re: Fine-tuning LLMs is a waste of time

#22

Post by lyu07282 »

sota_pop wrote: post #16
lyu07282 wrote: post #10

Earlier quoted context omitted.

> That said, fine tuning small models Mostly referred to as model distillation, but I give the author the benefit of the doubt that they didn't mean that.

My understanding of model distillation is quite different in that it trains another (typically smaller) model using the error between the new model’s output and that of the existing - effectively capturing the existing model’s embedded knowledge and encoding it (ideally more densely) into the new.

What what I was referring to is similar in concept, but I've seen both described in papers as distillation. What I meant was you take the output of a large model like GPT4 and use that as training data to fine-tune a smaller model.

Re: Fine-tuning LLMs is a waste of time

#23

Post by Mathnerd314 »

sota_pop wrote: post #14
Mathnerd314 wrote: post #13

Wasn't there that thing about how large LLM's are essentially compression algorithms ( https://arxiv.org/pdf/2309.10668 )? Maybe that's where this article is coming from, is the idea that finetuning "adds" data to the set of data that compresses well. But that indeed doesn't work unless you mix in the finetuning data with the original training corpus of the base model. I think the article is wrong though in saying it…

Not sure what you mean by “not trained to saturation”. Also I agree with the article, in the literature, the phenomenon to which the article refers is known as “catastrophic forgetting”. Because no one has specific knowledge about which weights contribute to model performance, by updating the weights via fine-tuning, you are modifying the model such that future performance will change in ways that are not understood.…

OK, so this intuition is actually a bit hard to unpack, I got it from bits and pieces. So this is this post https://www.fast.ai/posts/2023-09-04-learning-jumps/. Essentially, a single pass over the training data is enough for the LLM to significantly "learn" the material. In fact if you read the LLM training papers, for the large-large models, they generally explicitly say that they only did 1 pass over the training corpus, and sometimes not even the full corpus, only like 80% of it or whatever. The other relevant information is the loss curves - models like Llama 3 are not trained until the loss on the training data is minimized, like typical ML models. Rather they use these approximate estimates of FLOPS / tokens vs. performance on benchmarks. But it is pretty much guaranteed that if you continued to train on the training data it would continue to improve its fit - 1 pass over the training data is by no means enough to adequately learn all of the patterns. So from a compression standpoint, the paper I linked previously says that an LLM is a great compressor - but it's not even fully tuned, hence "not trained to saturation".

Now as far as how fine-tuning affects model performance, it is pretty simple: improves fit on the fine-tuning data, decreases fit on original training corpus. Beyond that, yeah, it is hard to say if fine-tuning will help you solve your problem. My experience has been that it always hurts generalization, so if you aren't getting reasonable results with a base or chat-tuned model, then fine-tuning further will not help, but if you are getting results then fine-tuning will make it more consistent.

Re: Fine-tuning LLMs is a waste of time

#24

Post by gdiamos »

me_vinayakakv wrote: post #18
gdiamos wrote: post #15

It's pretty frustrating to spend weeks on finetuning and end up with a model that says: "SELECT SELECT SELECT SELECT SELECT SELECT SELECT SELECT SELECT SELECT SELECT SELECT SELECT SELECT SELECT SELECT SELECT SELECT SELECT SELECT SELECT SELECT SELECT SELECT SELECT SELECT SELECT SELECT SELECT SELECT SELECT SELECT ..."

What is the way out in such cases? I've hit this with gemini-2.0-flash and changing the prompt ever so slightly seems to make things work, just to break it at other input.

Finetuning is deep learning training. It's pretty difficult to get right.

Andrej's 2019 blog laments on some of the reasons why it is hard and I can relate to a lot of this - https://karpathy.github.io/2019/04/25/recipe

The biggest mistake I see people making is this quote from the blog: "a 'fast and furious' approach to training neural networks does not work and only leads to suffering"

I'll probably write more about it in a few months...

Re: Fine-tuning LLMs is a waste of time

#26

Post by lovelearning »

cbsmith wrote: post #7
simonw wrote: post #6

"Fine-tuning large language models (LLMs) is frequently sold as a quick, powerful method for injecting new knowledge" Is that true though? I don't think I've seen a vendor selling that as a benefit of fine-tuning.

Yeah, as soon as I read that I felt like the author was living in a very different context from mine. It's never even occurred to me that fine-tuning could be an effective method for injecting new knowledge. If anything, I expect fine-tuning to destroy knowledge (and reasoning), which hopefully (if you did your fine-tuning right) is not relevant to the particular context you are fine-tuning for.

OpenAI makes statements like: [1]

1) "excel at a particular task"

2) "train on proprietary or sensitive data"

3) "Complex domain-specific tasks that require advanced reasoning", "Medical diagnosis based on history and diagnostic guidelines", "Determining relevant passages from legal case law"

4) "The general idea of fine-tuning is much like training a human in a particular subject, where you come up with the curriculum, then teach and test until the student excels."

Don't all these effectively inject new knowledge? It may happen through simultaneous destruction of some existing knowledge but that isn't obvious to non-technical people.

OpenAI's analogy of training a human in a particular subject until they excel even arguably excludes the possibility of destruction because we don't generally destroy existing knowledge in our minds to learn new things (but some of us may forget the older knowledge over time).

I'm a dev with hand-waving level of proficiency. I have fine-tuned self-hosted small LLMs using PyTorch. My perception of fine-tuning is that it fundamentally adds new knowledge. To what extent that involves destruction of existing knowledge has remained a bit vague.

My hand-waving solution if anyone pointed out that problem would be to 1) say that my fine-tuning data will include some of the foundational knowledge of the target subject to compensate for its destruction and 2) use a gold standard set of responses to verify the model after fine-tuning.

I for one found the article quite valuable for pointing out the problem and suggesting better approaches.

[1]: https://platform.openai.com/docs/guides/fine-tuning

Re: Fine-tuning LLMs is a waste of time

#27

Post by Nevermark »

kamranjon wrote: post #12

This is a pretty awful take. Everyone understands they are modifying the weights - that is the point. It’s not like these models were released with all of the weights perfectly accounted for and changing them in any way ruins them. The awesome thing about fine-tuning is that the weights are malleable and you have a great base to start from. Also the basic premise that knowledge injection is a bad use-case seems flawe…

> It’s not like these models were released with all of the weights perfectly accounted for and changing them in any way ruins them.

So more imperfect is better?

Of course the model’s parameters leave a many billions of elements vector path for improvement. But what circuitous path is that, which it didn’t already find?

You can’t find it by definition if you don’t include all the original data with the tuning data. You have radically changed the optimization surface with no contribution from the previous data at all.

The one use case that makes sense is sacrificing functionality to get better at a narrow problem.

You are correct about that.

Re: Fine-tuning LLMs is a waste of time

#28

Post by stingraycharles »

rybosome wrote: post #21

I think the point the author misses is that many applications of fine-tuning are to get a model to do a single task. This is what I have done in my current role at my company. We’ve fine-tuned open weight models for knowledge-injection, among other things, and get a model that’s better than OpenAI models at exactly one hyper specific task for our use case, which is hardware verification. Or, fine-tuned the OAI models…

Exactly. I want the LLM to be able to respond to our customers’ questions accurately and/or generate proper syntax for our query language.

The whole point of base models is to be general purpose, and fine tuned models to be tuned for specific tasks using a base model.

Re: Fine-tuning LLMs is a waste of time

#29

Post by 3abiton »

rybosome wrote: post #21

I think the point the author misses is that many applications of fine-tuning are to get a model to do a single task. This is what I have done in my current role at my company. We’ve fine-tuned open weight models for knowledge-injection, among other things, and get a model that’s better than OpenAI models at exactly one hyper specific task for our use case, which is hardware verification. Or, fine-tuned the OAI models…

Interestingly the author mentions LoRa as a "special" way for fine-tuning thatis not destructive. Have you considered it or you opted for more direct fine-tuning?

Re: Fine-tuning LLMs is a waste of time

#30

Post by ilaksh »

Obviously there are going to be narrow tasks where fine tuning makes sense. But using leading models for agents is a completely different mindset and approach.

Because I have been working on replacing multiple humans handling complex business processes mostly end-to-end (with human in the loop somehow in there).

I find that I need the very best models to be able to handle a lot of instructions and make the best decisions about tool selection. And overall I just need the most intelligence possible to make fewer weird errors or misinterpretations of the instructions or situations/data.

I can see how fine tuning would help for some issues like some report formatting. But that output comes at the end of the whole process. And I can address formatting issues almost instantly by either just using a smarter model that follows instructions better, or adding a reminder instruction, or creating a simpler subtask. Sometimes the subtask can run on a cheaper model.

So it's kind of like the difference between building a traditional manufacturing line with very specific robot arms, tooling and and conveyor belts, versus plugging in just a few different humanoid robots with assembly manuals and access to more general purposes tools on their belt. You used to always have to build the full traditional line. In many cases that doesn't necessarily make sense anymore.

Post reply on HN