Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Fine-tuning LLMs is a waste of time

codinginterviewsmadesimple.substack.com

Post reply on HN
Print view View original

51–60 of 94 posts

Re: Fine-tuning LLMs is a waste of time

#51

Post by kouteiheika »

> Adapter Modules and LoRA (Low-Rank Adaptation) insert new knowledge through specialized, isolated subnetworks, leaving existing neurons untouched. This is best for stuff like formatting, specific chains, etc- all of which don’t require a complete neural network update.

This highlights to me that the author doesn't know what they're talking about. LoRA does exactly the same thing as normal fine-tuning, it's just a trick to make it faster and/or be able to do it on lower end hardware. LoRA doesn't add "isolated subnetworks" - LoRA parameters are added to the original weights!

Here's the equation for the forward pass from the original paper[1]:

    h = W_{0} * x + ∆W * x = W_{0} * x + B * A * x
where "W_{0}" are the original weights and "B" and "A" (which give us "∆W_{x}" after they're multiplied) are the LoRA adapter. And if you've been paying attention it should also be obvious that, mathematically, you can merge your LoRA adapter into the original weights (by doing "W = W_{0} + ∆W") which most people do, or you could even create a LoRA adapter from a fully fine-tuned model by calculating "W - W_{0}" to get ∆W and then do SVD to recover B and A.

If you know what you're doing anything you can do with LoRA you can also do with full-finetuning, but better. It might be true that it's somewhat harder to "damage" a model by doing LoRA (because the parameter updates are fundamentally low rank due to the LoRA adapters being low rank), but that's a skill issue and not a fundamental property.

[1] -- https://arxiv.org/pdf/2106.09685

Re: Fine-tuning LLMs is a waste of time

#52

Post by zviugfd »

RoyTyrell wrote: post #50
rybosome wrote: post #21

I think the point the author misses is that many applications of fine-tuning are to get a model to do a single task. This is what I have done in my current role at my company. We’ve fine-tuned open weight models for knowledge-injection, among other things, and get a model that’s better than OpenAI models at exactly one hyper specific task for our use case, which is hardware verification. Or, fine-tuned the OAI models…

Let me preface by saying I'm not skeptical about your answer or think you're full of crap. Can you give me an example or two about a single task that you fine-tune for? Just trying to familiarize myself with more AI engineering tasks.

Any classification task. For example in search ranking, does a document contain the answer to this question?

Re: Fine-tuning LLMs is a waste of time

#53

Post by rickcarlino »

RoyTyrell wrote: post #50
rybosome wrote: post #21

I think the point the author misses is that many applications of fine-tuning are to get a model to do a single task. This is what I have done in my current role at my company. We’ve fine-tuned open weight models for knowledge-injection, among other things, and get a model that’s better than OpenAI models at exactly one hyper specific task for our use case, which is hardware verification. Or, fine-tuned the OAI models…

Let me preface by saying I'm not skeptical about your answer or think you're full of crap. Can you give me an example or two about a single task that you fine-tune for? Just trying to familiarize myself with more AI engineering tasks.

I used fine-tuning back in the day because GPT 3.5 struggled with the concept of determining if two sentences were equivalent or not. This was for grading language learning drills. It was a single skill for a specific task and I had lots of example data from thousands of spaced repetition quiz sessions. The base model struggled with the vague concept of “close enough” equivalence. Since that time, the state of the art has advanced to the point that I don’t need it anymore. I could probably do it to save some money but I’m pretty happy with GPT 4.1.

Re: Fine-tuning LLMs is a waste of time

#54

Post by MrLeap »

kouteiheika wrote: post #51

> Adapter Modules and LoRA (Low-Rank Adaptation) insert new knowledge through specialized, isolated subnetworks, leaving existing neurons untouched. This is best for stuff like formatting, specific chains, etc- all of which don’t require a complete neural network update. This highlights to me that the author doesn't know what they're talking about. LoRA does exactly the same thing as normal fine-tuning, it's just a t…

> that's a skill issue and not a fundamental property

This made me laugh.

You seem like you may know something I've been curious about.

I'm a shader author these days, haven't been a data scientist for a while, so it's going to distort my vocab.

Say you've got a trained neural network living in a 512x512 structured buffer. It's doing great, but you get a new video card with more memory so you can afford to migrate it to a 1024x1024. Is the state of the art way to retrain with the same data but bigger initial parameters, or are there other methods that smear the old weights over a larger space to get a leg up? Anything like this accelerate training time?

... can you up sample a language model like you can lowres anime profile pictures? I wonder what the made up words would be like.

Re: Fine-tuning LLMs is a waste of time

#55

Post by jjani »

BenGosub wrote: post #47
rybosome wrote: post #21

I think the point the author misses is that many applications of fine-tuning are to get a model to do a single task. This is what I have done in my current role at my company. We’ve fine-tuned open weight models for knowledge-injection, among other things, and get a model that’s better than OpenAI models at exactly one hyper specific task for our use case, which is hardware verification. Or, fine-tuned the OAI models…

In this case, for doing specific tasks, it makes much more sense to optimize the prompts and the whole flow with DSPy, instead of just fine tuning for each task.

That's only viable if the quality of the outputs can be automatically graded, reliably. GP's case sounds like one where that's probably possible, but for lots of specific tasks that isn't feasible, including the other ones he names:

> write poetry, give me advice on cooking, or translate to German

Re: Fine-tuning LLMs is a waste of time

#56

Post by kouteiheika »

MrLeap wrote: post #54
kouteiheika wrote: post #51

> Adapter Modules and LoRA (Low-Rank Adaptation) insert new knowledge through specialized, isolated subnetworks, leaving existing neurons untouched. This is best for stuff like formatting, specific chains, etc- all of which don’t require a complete neural network update. This highlights to me that the author doesn't know what they're talking about. LoRA does exactly the same thing as normal fine-tuning, it's just a t…

> that's a skill issue and not a fundamental property This made me laugh. You seem like you may know something I've been curious about. I'm a shader author these days, haven't been a data scientist for a while, so it's going to distort my vocab. Say you've got a trained neural network living in a 512x512 structured buffer. It's doing great, but you get a new video card with more memory so you can afford to migrate it…

In general this is of course an active area of research, but yes, you can do something that and people have done it successfully[1] by adding extra layers to an existing model and then continuing to train it.

You have to be careful about the "same data" part though; ideally you want to train once on unique data[2] as excessive duplication can harm the performance of the model[3], although if you have limited data a couple of training epochs might be safe and actually improve the performance of the model[4].

[1] -- https://arxiv.org/abs/2312.15166

[2] -- https://arxiv.org/abs/1906.06669

[3] -- https://arxiv.org/abs/2205.10487

[4] -- https://galactica.org/static/paper.pdf

Re: Fine-tuning LLMs is a waste of time

#57

Post by xpe »

kouteiheika wrote: post #51

> Adapter Modules and LoRA (Low-Rank Adaptation) insert new knowledge through specialized, isolated subnetworks, leaving existing neurons untouched. This is best for stuff like formatting, specific chains, etc- all of which don’t require a complete neural network update. This highlights to me that the author doesn't know what they're talking about. LoRA does exactly the same thing as normal fine-tuning, it's just a t…

> LoRA does exactly the same thing as normal fine-tuning

You wrote exactly so I'm going to say "no". To clarify what I mean: LoRA seeks to accomplish a similar goal as "vanilla" fine-tuning but with a different method (freezing existing model weights while adding adapter matrices that get added to the original). LoRA isn't exactly the same mathematically either; it is a low-rank approximation (as you know).

> LoRA doesn't add "isolated subnetworks"

If you think charitably, the author is right. LoRA weights are isolated in the sense that they are separate from the base model. See e.g. https://www.vellum.ai/blog/how-we-reduced-cost-of-a-fine-tun... "The end result is we now have a small adapter that can be added to the base model to achieve high performance on the target task. Swapping only the LoRA weights instead of all parameters allows cheaper switching between tasks. Multiple customized models can be created on one GPU and swapped in and out easily."

> you can merge your LoRA adapter into the original weights (by doing "W = W_{0} + ∆W") which most people do

Yes, one can do that. But on what basis do you say that "most people do"? Without having collected a sample of usage myself, I would just say this: there are many good reasons to not merge (e.g. see link above): less storage space if you have multiple adapters, easier to swap. On the other hand, if the extra adapter slows inference unacceptably, then don't.

> This highlights to me that the author doesn't know what they're talking about.

It seems to me you are being some combination of: uncharitable, overlooking another valid way of reading the text, being too quick to judge.

Re: Fine-tuning LLMs is a waste of time

#58

Post by MrLeap »

kouteiheika wrote: post #56
MrLeap wrote: post #54

Earlier quoted context omitted.

> that's a skill issue and not a fundamental property This made me laugh. You seem like you may know something I've been curious about. I'm a shader author these days, haven't been a data scientist for a while, so it's going to distort my vocab. Say you've got a trained neural network living in a 512x512 structured buffer. It's doing great, but you get a new video card with more memory so you can afford to migrate it…

In general this is of course an active area of research, but yes, you can do something that and people have done it successfully[1] by adding extra layers to an existing model and then continuing to train it. You have to be careful about the "same data" part though; ideally you want to train once on unique data[2] as excessive duplication can harm the performance of the model[3], although if you have limited data a c…

Thank you for taking the time to provide me all this reading.

Re: Fine-tuning LLMs is a waste of time

#59

Post by kouteiheika »

xpe wrote: post #57
kouteiheika wrote: post #51

> Adapter Modules and LoRA (Low-Rank Adaptation) insert new knowledge through specialized, isolated subnetworks, leaving existing neurons untouched. This is best for stuff like formatting, specific chains, etc- all of which don’t require a complete neural network update. This highlights to me that the author doesn't know what they're talking about. LoRA does exactly the same thing as normal fine-tuning, it's just a t…

> LoRA does exactly the same thing as normal fine-tuning You wrote exactly so I'm going to say "no". To clarify what I mean: LoRA seeks to accomplish a similar goal as "vanilla" fine-tuning but with a different method (freezing existing model weights while adding adapter matrices that get added to the original). LoRA isn't exactly the same mathematically either; it is a low-rank approximation (as you know). > LoRA do…

> You wrote exactly so I'm going to say "no". [...] If you think charitably, the author is right.

No, the author is objectively wrong. Let me quote the article and clarify myself:

> Fine-tuning advanced LLMs isn’t knowledge injection — it’s destructive overwriting. [...] When you fine-tune, you risk erasing valuable existing patterns, leading to unexpected and problematic downstream effects. [...] Instead, use modular methods like [...] adapters.

This is just incorrect. LoRA is exactly like normal fine-tuning here in this particular context. The author's argument is that you should do LoRA because it doesn't do any "destructive overwriting", but in that aspect it's no different than normal fine-tuning.

In fact, there's evidence that LoRA can actually make the problem worse[1]:

> we first show that the weight matrices trained with LoRA have new, high-ranking singular vectors, which we call intruder dimensions [...] LoRA fine-tuned models with intruder dimensions are inferior to fully fine-tuned models outside the adaptation task’s distribution, despite matching accuracy in distribution.

[1] -- https://arxiv.org/pdf/2410.21228

To be fair, "if you don't know what you're doing then doing LoRA over normal finetuning" is, in general, a good advice in my opinion. But that's not what the article is saying.

> But on what basis do you say that "most people do"?

On the basis of seeing what the common practice is, at least in the open (in the local LLM community and in the research space).

> I would just say this: there are many good reasons to not merge

I never said that there aren't good reasons to not merge.

> It seems to me you are being some combination of: uncharitable, overlooking another valid way of reading the text, being too quick to judge.

No, I'm just tired of constantly seeing a torrent of misinformation from people who don't know much about how these models actually work nor have done any significant work on their internals, yet try to write about them with authority.

Re: Fine-tuning LLMs is a waste of time

#60

Post by bicepjai »

kouteiheika wrote: post #59
xpe wrote: post #57

Earlier quoted context omitted.

> LoRA does exactly the same thing as normal fine-tuning You wrote exactly so I'm going to say "no". To clarify what I mean: LoRA seeks to accomplish a similar goal as "vanilla" fine-tuning but with a different method (freezing existing model weights while adding adapter matrices that get added to the original). LoRA isn't exactly the same mathematically either; it is a low-rank approximation (as you know). > LoRA do…

> You wrote exactly so I'm going to say "no". [...] If you think charitably, the author is right. No, the author is objectively wrong. Let me quote the article and clarify myself: > Fine-tuning advanced LLMs isn’t knowledge injection — it’s destructive overwriting. [...] When you fine-tune, you risk erasing valuable existing patterns, leading to unexpected and problematic downstream effects. [...] Instead, use modula…

I learned a lot of perspective on LORA. Thanks folks
Post reply on HN