51–60 of 62 posts
Question for dudes building modern nn's... what's the thinking on estimating structural capacity for real world problem? How should I estimate how many parameters to choose for the model?
> However, the literature is unclear on how well LoRA performs relative to FullFT. I think the literature is clear on that? "LoRA vs Full Fine-tuning: An Illusion of Equivalence" -- https://arxiv.org/abs/2410.21228v1 Quoting from the conclusions: > The paper describes the finding that LoRA and full fine-tuning, with equal performance on the fine-tuning task, can have solutions with very different generalization behav…
> I'm surprised they didn't cite this; it's a well known paper. I'm surprised you copied and pasted all of that without explaining what it means. Does LoRA perform worse, better or statistically insignificantly different to FullFT? You aren't able to tell from what you pasted, are you?
There are techniques like PiCa and SVFT which can mitigate much of the loss, though.
Earlier quoted context omitted.
> I'm surprised they didn't cite this; it's a well known paper. I'm surprised you copied and pasted all of that without explaining what it means. Does LoRA perform worse, better or statistically insignificantly different to FullFT? You aren't able to tell from what you pasted, are you?
If you're going to be snarky, could you at least clarify what the answer is for those of us who don't stay on top of ML research...?
The answer is "There's a difference, perhaps", but the GP appeared to imply that LoRA performed worse.
My understanding is that that paper found differences, but did not conclude that the differences were quantifiably better or worse, but this is not what GP's post implied.
Stumbled on this today... https://hackerpager.net/
I really want something like this with flip out keyboard and could do Signal on LTE/WiFi.
Question for dudes building modern nn's... what's the thinking on estimating structural capacity for real world problem? How should I estimate how many parameters to choose for the model?
You test different models on your real world problem, and pick the smallest one that works.
Earlier quoted context omitted.
You test different models on your real world problem, and pick the smallest one that works.
I just think that there has to be some heuristic..
For example, for classification, if is hallucinating semantically similar, but not technically valid classes, you can probably fine-tune your way out of the gap with a smaller model.
But if your task requires world knowledge, you likely need a larger model. It's not cheap, efficient, or generally useful to fine-tune for additional world knowledge directly.
Earlier quoted context omitted.
> I'm surprised they didn't cite this; it's a well known paper. I'm surprised you copied and pasted all of that without explaining what it means. Does LoRA perform worse, better or statistically insignificantly different to FullFT? You aren't able to tell from what you pasted, are you?
Standard LoRA (W_delta = B@A with standard inits) generally underperforms FT, primarily because of "intruder dimensions" (new high-ranking singular vectors which misalign with the singular vectors of the underlying weights) as outlined in the paper. There are techniques like PiCa and SVFT which can mitigate much of the loss, though.