Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

LoRA Speedrun – a public wall-clock leaderboard for fine-tuning techniques

github.com

Post reply on HN
Print view View original

31–39 of 39 posts

Re: LoRA Speedrun – a public wall-clock leaderboard for fine-tuning techniques

#31

Post by nomel »

monegator wrote: post #12

Feels like i'm missing the introduction paragraph in the repo. What is LoRA in this context? the communication protocol? Or another term appropriated by LLMs? Why the speedrun?

The author responds with reasoning here: https://news.ycombinator.com/item?id=48975473

The AI slop "why" section in the readme is completely useless. I struggle with this too. I start every project doc with a "why" then "overview" section, and AI is still horrendously bad at the "why", filled with vapid corpo/tech startup speak. I usually write the "why" myself, or at least put down the outline to be completed.

Re: LoRA Speedrun – a public wall-clock leaderboard for fine-tuning techniques

#32

Post by nomel »

Vineeth147 wrote: post #7
stephantul wrote: post #6

Earlier quoted context omitted.

Not the software, the whole thing. As an author: show me why you thought this was interesting and why you’re doing it, and why you think it’s relevant. What does it build towards? What does climbing this leaderboard mean to me? Absent those things, this is just some thing my opus could generate as well.

I fine-tune small models with limited resources, but it's hard to tell which speedup claims are real. Each method, like DoRA, rsLoRA, Unsloth, or packing tricks, uses different models, data, and hardware in their reports. This makes it impossible to really check their claims in practice. The only way I've seen these debates settled is by using a fixed task and having someone act as a referee. That's what nanoGPT's sp…

Put something like this in the readme, ffs.

Re: LoRA Speedrun – a public wall-clock leaderboard for fine-tuning techniques

#33

Post by rsfern »

curiouscube wrote: post #28
rsfern wrote: post #21

Earlier quoted context omitted.

I don’t think there’s a fundamental reason that performance has to be monotonic in model size or even training FLOPs. At least I don’t think it’s been proved to be so, so I think “misinformed” is a bit premature and sort of makes GP’s point. There’s evidence that model size and representational capacity are not exactly the same, and that scale is maybe more important for learning than it is for representation (past a…

I'm not saying that it is impossible for a more advanced model and training paradigm to outperform a larger model, what I'm saying is that if you leave everything the same except model size, the larger model nearly always outperforms the smaller one. This is intuitively true if you consider that you can fit in the smaller model + extra parameters into a larger model. There are some cases in which it doesn't work due…

My point with the force field example wasn’t to argue against neural scaling as a valid strategy, it totally is effective and a lot of groups are doing it. But I feel like we might be talking past each other a bit.

What I’m pushing back on is what I think is a sort of one-dimensional view of Sutton’s bitter lesson. People seem to equate it with model scaling, but there are lots of general ways to leverage computation that don’t involve just scaling models and supervised training datasets up. For example Sutton’s first example is straight up search, no parameters at all.

The point of the force field example is that it seems you don’t need billions of parameters to represent the functions we’re interested in, but with small models it’s harder to find those functions by pushing harder on the standard training algorithms, and that maybe some different algorithm that leverages computation more effectively could do so.

Re: LoRA Speedrun – a public wall-clock leaderboard for fine-tuning techniques

#34

Post by Vineeth147 »

nomel wrote: post #32
Vineeth147 wrote: post #7

Earlier quoted context omitted.

I fine-tune small models with limited resources, but it's hard to tell which speedup claims are real. Each method, like DoRA, rsLoRA, Unsloth, or packing tricks, uses different models, data, and hardware in their reports. This makes it impossible to really check their claims in practice. The only way I've seen these debates settled is by using a fixed task and having someone act as a referee. That's what nanoGPT's sp…

Put something like this in the readme, ffs.

Done — the why section is now basically that comment, first person, no filler.

Re: LoRA Speedrun – a public wall-clock leaderboard for fine-tuning techniques

#36

Post by Vineeth147 »

patrick0d wrote: post #23

Inspired by parameter golf and speedrun approaches I make the case for picking loss functions like a wallclock for LoRA on AI safety targets. The result when I tried it was a functional distillation of an Sparse AutoEncoder into a 5.3MB probe. I have a technical writeup below about it if anyone is interested. https://www.lesswrong.com/posts/PagGF8roBJmjLunsX/competitiv...

[dead]

Re: LoRA Speedrun – a public wall-clock leaderboard for fine-tuning techniques

#37

Post by Vineeth147 »

Aerialoo wrote: post #13
Vineeth147 wrote: post #4

Earlier quoted context omitted.

That's fair on both points. Much of this was built with AI, but the runs and numbers are real. They are also reproducible, so I would prefer to be judged on that. And yes, using a single model and task can lead to overfitting. The plan is to add more tracks, including bigger models and other tasks, so a technique only matters if it transfers. Right now, it's just the initial track, so your concern is valid. Thanks fo…

AI generates or not - irrelevant if the substance is high quality. Always good to see projects. You should consider hosting this leaderboard on hugging face, you will get a lot more AI enthusiasts and practitioners there.

[dead]

Re: LoRA Speedrun – a public wall-clock leaderboard for fine-tuning techniques

#38

Post by Vineeth147 »

jmward01 wrote: post #11

I think there is real value in going smaller/limiting resources. The trend is 'just make the weights bigger and throw more data at it'. It is a MBA's view of winning. We have a knob, keep turning it. It does work but it may not drive as much creativity as resource limits can drive. It is like urban growth boundaries in city planning. If you aren't allowed to 'just expand' you are forced to build more intelligently in…

[flagged]

Re: LoRA Speedrun – a public wall-clock leaderboard for fine-tuning techniques

#39

Post by curiouscube »

rsfern wrote: post #33
curiouscube wrote: post #28

Earlier quoted context omitted.

I'm not saying that it is impossible for a more advanced model and training paradigm to outperform a larger model, what I'm saying is that if you leave everything the same except model size, the larger model nearly always outperforms the smaller one. This is intuitively true if you consider that you can fit in the smaller model + extra parameters into a larger model. There are some cases in which it doesn't work due…

My point with the force field example wasn’t to argue against neural scaling as a valid strategy, it totally is effective and a lot of groups are doing it. But I feel like we might be talking past each other a bit. What I’m pushing back on is what I think is a sort of one-dimensional view of Sutton’s bitter lesson. People seem to equate it with model scaling, but there are lots of general ways to leverage computation…

I totally agree with what you're saying. Your point about other methods to leverage computation is true and important.

But that's exactly my claim: Smaller models can win on the computational efficiency front, but not the overall capability front. And as long as compute is getting cheaper, investing in more efficiency while there are still major capability on the table isn't a good business strategy.

Smaller, efficient models could lead to some really interesting things though especially considering it could lead to some Jevon's paradox like moment. To be honest, I feel like the biggest issue in regards to LLM usage in practice is that the patterns of use aren't really well developed. Yes we have agents, but it's still somewhat unclear what an agent can "do" - People seem to mostly focus on replacing some kind of existing process with an agent driven one, but actually coming up with AI-native processes is way harder.

Post reply on HN