Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

LoRA Learns Less and Forgets Less

arxiv.org

Post reply on HN
Print view View original

31–40 of 65 posts

Re: LoRA Learns Less and Forgets Less

#31

Post by nerdponx »

Findecanor wrote: post #30
dheera wrote: post #27

Earlier quoted context omitted.

Not sure about "popular" 99% of ML engineers wouldn't know what it is.

99% of the engineers who are still working in ML (Machine Language) would. A much smaller percent among those who write in ML (the functional programming language) probably, though.

Even if the ML engineers know about the wireless protocol (and I doubt that many do), the scientists/researchers who develop these models probably don't. They are completely different domain. The lead author on this paper is basically a neuroscientist; some of the other are technically computer scientists, but probably have little hands-on experience with networking beyond whatever they did in undergrad.

Re: LoRA Learns Less and Forgets Less

#32

Post by sva_ »

squarefoot wrote: post #12
sva_ wrote: post #11

Earlier quoted context omitted.

Yes, but this is LoRA, clearly not LoRa.

PP has a point though. I entered "LoRA" on Google, DuckDuckGo, Startpage and Bing, and all returned results in all first pages were about the communication protocol (1). They could have inferred my interests from previous searches, but I never used Bing in the last year or so, so it seems to me someone didn't care about name clashes. (1) well, except Google which -surprise- returned about mid page an ad of a local qu…

I usually just add a term like 'ml' or 'nn' after my search to give the machine context and it is sufficient in most cases.

Re: LoRA Learns Less and Forgets Less

#33

Post by rzzzt »

This paper has 12 authors, which fascinates me to no end for some unexplainable reason. How does it work? Is it a common occurrence to have this many people working on a submission? Did each of them get at least a paragraph in edgewise?

Re: LoRA Learns Less and Forgets Less

#35

Post by repsak »

rzzzt wrote: post #33

This paper has 12 authors, which fascinates me to no end for some unexplainable reason. How does it work? Is it a common occurrence to have this many people working on a submission? Did each of them get at least a paragraph in edgewise?

I raise you the Gemini paper https://arxiv.org/abs/2312.11805

Re: LoRA Learns Less and Forgets Less

#36

Post by goodpoint »

dheera wrote: post #27
thepasswordis wrote: post #10

I really wish people would be more careful about choosing names for these things. LoRa has been a popular wireless protocol for like 10 years.

Not sure about "popular" 99% of ML engineers wouldn't know what it is.

...but they should know how to use search engines...

Re: LoRA Learns Less and Forgets Less

#37

Post by guyomes »

repsak wrote: post #35
rzzzt wrote: post #33

This paper has 12 authors, which fascinates me to no end for some unexplainable reason. How does it work? Is it a common occurrence to have this many people working on a submission? Did each of them get at least a paragraph in edgewise?

I raise you the Gemini paper https://arxiv.org/abs/2312.11805

All-in with the Foldit paper [1,2].

[1]: https://en.wikipedia.org/wiki/Foldit

[2]: https://www.nature.com/articles/nature09304

Re: LoRA Learns Less and Forgets Less

#38

Post by SubiculumCode »

rzzzt wrote: post #33

This paper has 12 authors, which fascinates me to no end for some unexplainable reason. How does it work? Is it a common occurrence to have this many people working on a submission? Did each of them get at least a paragraph in edgewise?

For a serious answer, this is how it works in my field A researcher gets a grant with 3-7 co-investigators. This generates a bunch of data and other resources that will support 10 or more papers. Coinvestigators and PIs will ask their postdocs and grad students to write up a paper. PIs and co-Is go on every paper...because it's a paper from their grant. Then the 1 to 4 grad students and post-docs go on the paper, depending on their specific material contributions to the work, be it analysis, conception, or execution, or writing. The numbers can stack up.

Re: LoRA Learns Less and Forgets Less

#39

Post by PeterisP »

rzzzt wrote: post #33

This paper has 12 authors, which fascinates me to no end for some unexplainable reason. How does it work? Is it a common occurrence to have this many people working on a submission? Did each of them get at least a paragraph in edgewise?

The general criteria for authorship require including the people who worked on the experiments and data for the paper, which can be more important contribution than most of the text in that paper. In other experimental fields, there are papers with dozens or even hundreds of authors, because it can take many people to get to a measurement of a single number in the paper.

Re: LoRA Learns Less and Forgets Less

#40

Post by 3abiton »

danielhanchen wrote: post #19
chaos_emergent wrote: post #6

The findings are that the best fine-tune performance comes from fine-tuning all weights, followed my MLPs, followed by attention heads, using LoRA. Authors assert that the performance difference is based on the target module of the NN. Isn’t an equally valid argument that MLPs tend to constitute a greater number of weights in transformer networks than attention heads, and the performance difference can be traced to a…

I think the QLoRA paper https://arxiv.org/pdf/2305.14314 paper also showed LoRA on all MLP + Attention layers > all MLP layers > just Attention layers. Other papers show finetuning a select few layers can also work well.

Any real world performance comparison between QLoRa and LoRa?
Post reply on HN