Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Fine-tuning LLMs is a waste of time

codinginterviewsmadesimple.substack.com

Post reply on HN
Print view View original

61–70 of 94 posts

Re: Fine-tuning LLMs is a waste of time

#61

Post by jahewson »

xpe wrote: post #57
kouteiheika wrote: post #51

> Adapter Modules and LoRA (Low-Rank Adaptation) insert new knowledge through specialized, isolated subnetworks, leaving existing neurons untouched. This is best for stuff like formatting, specific chains, etc- all of which don’t require a complete neural network update. This highlights to me that the author doesn't know what they're talking about. LoRA does exactly the same thing as normal fine-tuning, it's just a t…

> LoRA does exactly the same thing as normal fine-tuning You wrote exactly so I'm going to say "no". To clarify what I mean: LoRA seeks to accomplish a similar goal as "vanilla" fine-tuning but with a different method (freezing existing model weights while adding adapter matrices that get added to the original). LoRA isn't exactly the same mathematically either; it is a low-rank approximation (as you know). > LoRA do…

Sorry to be a downer but basically every statement you’ve made above is incorrect.

Re: Fine-tuning LLMs is a waste of time

#62

Post by freehorse »

rybosome wrote: post #21

I think the point the author misses is that many applications of fine-tuning are to get a model to do a single task. This is what I have done in my current role at my company. We’ve fine-tuned open weight models for knowledge-injection, among other things, and get a model that’s better than OpenAI models at exactly one hyper specific task for our use case, which is hardware verification. Or, fine-tuned the OAI models…

The author makes it specific they talk about finetuning "for Knowledge Injection". The give a quote that claims that finetuning is still useful for things like following a specific style, formatting etc. The title they chose could have been a bit more specific and less aphoristic.

What finetuning makes less sense is doing it merely to get a model eg up to date with changes in some library, or to learn a new library it did not know, or, even worse, your codebase. I think this is what OP talks about.

Re: Fine-tuning LLMs is a waste of time

#63

Post by gwd »

rybosome wrote: post #21

I think the point the author misses is that many applications of fine-tuning are to get a model to do a single task. This is what I have done in my current role at my company. We’ve fine-tuned open weight models for knowledge-injection, among other things, and get a model that’s better than OpenAI models at exactly one hyper specific task for our use case, which is hardware verification. Or, fine-tuned the OAI models…

> I think the point the author misses...

It looked to me like the author did know that. The title only says "Fine-tuning", but immediately in the article he talks about Fine-tuning for knowledge injection, in order to "ensure that their systems were always updated with new information".

Fine-tuning to help it not make the stupid mistake that it makes 10% of the time no matter what instructions you give it is a completely different use case.

Re: Fine-tuning LLMs is a waste of time

#64

Post by arbfay »

Before post-ChatGPT boom, we used to talk of "catastrophic forgetting"...

Make sure the new training dataset is "large" by augmenting it with general data (see it as a sample of the original dataset), use PEFT techniques (freezing weights => less risks), use regularization (elastic weight consolidation).

Fine-tuning is fine, but will be more expensive that you thought and should be led by more experienced ML engineers. You probably don't need to fine tune models anyway.

Re: Fine-tuning LLMs is a waste of time

#65

Post by yorwba »

kouteiheika wrote: post #56
MrLeap wrote: post #54

Earlier quoted context omitted.

> that's a skill issue and not a fundamental property This made me laugh. You seem like you may know something I've been curious about. I'm a shader author these days, haven't been a data scientist for a while, so it's going to distort my vocab. Say you've got a trained neural network living in a 512x512 structured buffer. It's doing great, but you get a new video card with more memory so you can afford to migrate it…

In general this is of course an active area of research, but yes, you can do something that and people have done it successfully[1] by adding extra layers to an existing model and then continuing to train it. You have to be careful about the "same data" part though; ideally you want to train once on unique data[2] as excessive duplication can harm the performance of the model[3], although if you have limited data a c…

In addition to increasing the number of layers, you can also grow the weight matrices and initialize by tiling them with the smaller model's weights https://neurips.cc/media/neurips-2023/Slides/83968_5GxuY2z.p...

Re: Fine-tuning LLMs is a waste of time

#66

Post by ijk »

kouteiheika wrote: post #56
MrLeap wrote: post #54

Earlier quoted context omitted.

> that's a skill issue and not a fundamental property This made me laugh. You seem like you may know something I've been curious about. I'm a shader author these days, haven't been a data scientist for a while, so it's going to distort my vocab. Say you've got a trained neural network living in a 512x512 structured buffer. It's doing great, but you get a new video card with more memory so you can afford to migrate it…

In general this is of course an active area of research, but yes, you can do something that and people have done it successfully[1] by adding extra layers to an existing model and then continuing to train it. You have to be careful about the "same data" part though; ideally you want to train once on unique data[2] as excessive duplication can harm the performance of the model[3], although if you have limited data a c…

This might be obvious, but just to state it explicitly for everyone: you can freeze the weights of the existing layers if you want to train the new layers but want to leave the existing layers untouched.

Re: Fine-tuning LLMs is a waste of time

#67

Post by BenGosub »

ericflo wrote: post #49
BenGosub wrote: post #47

Earlier quoted context omitted.

In this case, for doing specific tasks, it makes much more sense to optimize the prompts and the whole flow with DSPy, instead of just fine tuning for each task.

It's not either/or. Generally you finetune when optimized many-shot still doesn't hit your desired quality bar. And it turns out with RL, things like system prompts matter a lot, so searching over prompts is a good idea even when reinforcing the desirable circuits.

I am not an expert in fine tuning, but in the company I work for our fine tuned model didn't do any noticeable difference.

Re: Fine-tuning LLMs is a waste of time

#68

Post by BenGosub »

jjani wrote: post #55
BenGosub wrote: post #47

Earlier quoted context omitted.

In this case, for doing specific tasks, it makes much more sense to optimize the prompts and the whole flow with DSPy, instead of just fine tuning for each task.

That's only viable if the quality of the outputs can be automatically graded, reliably. GP's case sounds like one where that's probably possible, but for lots of specific tasks that isn't feasible, including the other ones he names: > write poetry, give me advice on cooking, or translate to German

Certainly, in those cases one needs to be clever and design an evaluation framework that will grade based on soft criteria, or maybe use user feedback. Still, over time a good train-test database should be built and leveraging dspy will do improvements even in those cases.

Re: Fine-tuning LLMs is a waste of time

#70

Post by [deleted] »

xpe wrote: post #57
kouteiheika wrote: post #51

> Adapter Modules and LoRA (Low-Rank Adaptation) insert new knowledge through specialized, isolated subnetworks, leaving existing neurons untouched. This is best for stuff like formatting, specific chains, etc- all of which don’t require a complete neural network update. This highlights to me that the author doesn't know what they're talking about. LoRA does exactly the same thing as normal fine-tuning, it's just a t…

> LoRA does exactly the same thing as normal fine-tuning You wrote exactly so I'm going to say "no". To clarify what I mean: LoRA seeks to accomplish a similar goal as "vanilla" fine-tuning but with a different method (freezing existing model weights while adding adapter matrices that get added to the original). LoRA isn't exactly the same mathematically either; it is a low-rank approximation (as you know). > LoRA do…

[deleted]
Post reply on HN