31–40 of 62 posts
> LoRA works well when not capacity constrained, i.e., the number of trainable parameters exceeds the amount of information to be learned, which can be estimated in terms of dataset size I’m shocked they didn’t look at progressive merging of LoRAs. Research shows that’s the best way of improving its ability to model higher level features. Seems like a massive miss, not to mention there is other research that contradi…
> LoRA works well when not capacity constrained, i.e., the number of trainable parameters exceeds the amount of information to be learned, which can be estimated in terms of dataset size I’m shocked they didn’t look at progressive merging of LoRAs. Research shows that’s the best way of improving its ability to model higher level features. Seems like a massive miss, not to mention there is other research that contradi…
I am curious, would you mind sharing a citation?
> However, the literature is unclear on how well LoRA performs relative to FullFT. I think the literature is clear on that? "LoRA vs Full Fine-tuning: An Illusion of Equivalence" -- https://arxiv.org/abs/2410.21228v1 Quoting from the conclusions: > The paper describes the finding that LoRA and full fine-tuning, with equal performance on the fine-tuning task, can have solutions with very different generalization behav…
I'm surprised you copied and pasted all of that without explaining what it means.
Does LoRA perform worse, better or statistically insignificantly different to FullFT?
You aren't able to tell from what you pasted, are you?
> However, the literature is unclear on how well LoRA performs relative to FullFT. I think the literature is clear on that? "LoRA vs Full Fine-tuning: An Illusion of Equivalence" -- https://arxiv.org/abs/2410.21228v1 Quoting from the conclusions: > The paper describes the finding that LoRA and full fine-tuning, with equal performance on the fine-tuning task, can have solutions with very different generalization behav…
To say that the 'literature is clear on that' while citing a single paper, which has been rejected from ICLR, is a bit of an overstatement.
Oh, you mean rejected just like these papers?
Efficient Estimation of Word Representations in Vector Space[1], one of the most influential papers in the space with tens of thousands of citations[2]? Or the RoBERTa[3] paper (dramatically improved upon BERT; RoBERTa and derived models currently have tens of millions of downloads on HF and still serve as a reliable industry workhorse)? Or the Mamba paper[4] (pretty much the only alternative to transformers that actually gets used)? Do you want me to keep going?
Honestly, I find that whether a paper gets rejected or not means diddly squat considering how broken the review system is, and through how much honestly terrible papers I have to wade through every time I'm looking through the conference submissions for anything good.
[1] -- https://openreview.net/forum?id=idpCdOWtqXd60
[2] -- https://scholar.google.com/scholar?cites=7447715766504981253
The name gets me every single time. Always think it’s going to be about radio LoRa
Both are very cool, but I wonder if I missed something else?
The name gets me every single time. Always think it’s going to be about radio LoRa
Earlier quoted context omitted.
Not just me then. It's always the first thing that springs to mind.
Nope not just you! Gets me everytime.
At this point I think they do it on purpose, as their metrics for "people visiting the website/repository" or whatever gets increased as people thinking the repository is about the existing concept/technology.
I've been curious about LoRA and find a lot of these articles interesting. But I've been unable to find a good "LoRA for idiots" kind of starting point that gets me started actually doing some training with my data. Anybody know of a more practical guide I could use for that?
Be sure to validate everything you're reading though as of late I've come across more and more things that don't seem 100% accurate in their docs, seems to heavily depend on what section.
Can someone explain the bit counting argument in the reinforcement learning part? I don’t get why a trajectory would provide only one bit of information. Each step of the trajectory is at least giving information about what state transitions are possible. An infinitely long trajectory can explore the whole state space if there are no absorbing states. Such a trajectory would provide a massive amount of information ab…
This is in contrast to more "supervised" forms of learning where you could get a loss for each token produced (e.g. cross entropy loss), and where you'd get, as a consequence O(number of tokens) information into your gradients.