Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

LoRA Speedrun – a public wall-clock leaderboard for fine-tuning techniques

github.com

Post reply on HN
Print view View original

21–30 of 39 posts

Re: LoRA Speedrun – a public wall-clock leaderboard for fine-tuning techniques

#21

Post by rsfern »

curiouscube wrote: post #16
jmward01 wrote: post #11

I think there is real value in going smaller/limiting resources. The trend is 'just make the weights bigger and throw more data at it'. It is a MBA's view of winning. We have a knob, keep turning it. It does work but it may not drive as much creativity as resource limits can drive. It is like urban growth boundaries in city planning. If you aren't allowed to 'just expand' you are forced to build more intelligently in…

You are right only in so far that it is more economical. But it is not the MBA's view of winning, it's just one potential conclusion you could draw from the bitter lesson of Machine Learning. As long as the need for more intelligence outpaces the economics of using intelligence, you'll get bigger models. This idea that small, fine-tuned models can outperform bigger models capabilities wise is mostly misinformed. They…

I don’t think there’s a fundamental reason that performance has to be monotonic in model size or even training FLOPs. At least I don’t think it’s been proved to be so, so I think “misinformed” is a bit premature and sort of makes GP’s point.

There’s evidence that model size and representational capacity are not exactly the same, and that scale is maybe more important for learning than it is for representation (past a point). Consider the early work from the current neural scaling paradigm. The Chinchilla scaling study shows that smaller models can match the performance of larger models by training longer.

To GP’s point, if everyone is exploiting the scaling lever, few resources are being allocated to finding more efficient training algorithms that could let us work with right-sized models instead of pulling the scaling lever as hard as we can afford to.

I’ll end with a dramatic example from my field of materials science (which admittedly might not strictly generalize to LLMs). A lot of the field is pursuing the model scaling strategy, and it’s still paying off. But [0] recently reported competitive accuracy with much smaller models that run faster and can address much larger problems. The model architecture is pretty much the same, but they use a different training strategy and really focus on data quality

0: https://arxiv.org/abs/2504.21286

Re: LoRA Speedrun – a public wall-clock leaderboard for fine-tuning techniques

#22

Post by garimbaboy »

monegator wrote: post #12

Feels like i'm missing the introduction paragraph in the repo. What is LoRA in this context? the communication protocol? Or another term appropriated by LLMs? Why the speedrun?

LoRa (Long Range) is a physical radio communication techniche.[1]

LoRA (Low-Rank Adaptation) is a parameter-efficient fine-tuning technique for large language models.[2]

[1] https://en.wikipedia.org/wiki/LoRa

[2] https://en.wikipedia.org/wiki/LoRA_(machine_learning)

Re: LoRA Speedrun – a public wall-clock leaderboard for fine-tuning techniques

#23

Post by patrick0d »

Inspired by parameter golf and speedrun approaches I make the case for picking loss functions like a wallclock for LoRA on AI safety targets. The result when I tried it was a functional distillation of an Sparse AutoEncoder into a 5.3MB probe. I have a technical writeup below about it if anyone is interested.

https://www.lesswrong.com/posts/PagGF8roBJmjLunsX/competitiv...

Re: LoRA Speedrun – a public wall-clock leaderboard for fine-tuning techniques

#25

Post by walrus01 »

Reading the title, for a moment I thought this would be something about a LoRA serial data bridge over RF and an actual physical wall clock, possibly some novel new homebuilt piece of hardware to display perfectly accurate NTP synchronized time or something.

Re: LoRA Speedrun – a public wall-clock leaderboard for fine-tuning techniques

#27

Post by rocqua »

rsfern wrote: post #21
curiouscube wrote: post #16

Earlier quoted context omitted.

You are right only in so far that it is more economical. But it is not the MBA's view of winning, it's just one potential conclusion you could draw from the bitter lesson of Machine Learning. As long as the need for more intelligence outpaces the economics of using intelligence, you'll get bigger models. This idea that small, fine-tuned models can outperform bigger models capabilities wise is mostly misinformed. They…

I don’t think there’s a fundamental reason that performance has to be monotonic in model size or even training FLOPs. At least I don’t think it’s been proved to be so, so I think “misinformed” is a bit premature and sort of makes GP’s point. There’s evidence that model size and representational capacity are not exactly the same, and that scale is maybe more important for learning than it is for representation (past a…

I especially wonder about the quality of N big models in a delegation harness against M smaller models in a delegation harness with N and M tuned to use the same amount of total compute. (So a few agents of big models against a lot of big models). I wouldn't be surprised if the advantages of delegation and the corresponding compression of context outweighs the lower quality of the smaller models.

Or put differently a wide exploration of long chains of obvious insights might be more valuable than a more narrow exploration of shorter chains of deeper insights. But perhaps that is a wrong sense of the difference between a small model and a large model.

Re: LoRA Speedrun – a public wall-clock leaderboard for fine-tuning techniques

#28

Post by curiouscube »

rsfern wrote: post #21
curiouscube wrote: post #16

Earlier quoted context omitted.

You are right only in so far that it is more economical. But it is not the MBA's view of winning, it's just one potential conclusion you could draw from the bitter lesson of Machine Learning. As long as the need for more intelligence outpaces the economics of using intelligence, you'll get bigger models. This idea that small, fine-tuned models can outperform bigger models capabilities wise is mostly misinformed. They…

I don’t think there’s a fundamental reason that performance has to be monotonic in model size or even training FLOPs. At least I don’t think it’s been proved to be so, so I think “misinformed” is a bit premature and sort of makes GP’s point. There’s evidence that model size and representational capacity are not exactly the same, and that scale is maybe more important for learning than it is for representation (past a…

I'm not saying that it is impossible for a more advanced model and training paradigm to outperform a larger model, what I'm saying is that if you leave everything the same except model size, the larger model nearly always outperforms the smaller one. This is intuitively true if you consider that you can fit in the smaller model + extra parameters into a larger model. There are some cases in which it doesn't work due to dataset/model size incompatibilities leading to overfitting, this is mostly covered by the neural scaling laws.

Based on a quick cursory glance at your example: Better data + better technique led to a better result with less parameters. Would you assume that then scaling both the dataset and model once again would lead to even better results? If you haven't fully encompassed the underlying distribution with datapoints, intuition says yes.

What I wanted to initially highlight was actually something slightly different: Specifically that people keep trying to "outsmart" optimizers by either fully hand-crafting solutions or skewing existing machine learning algorithms via additional tricks that are supposed to encode "human intuition" or something similar (to be fair there are ways to do it correctly). These all tend to fall short in a few years simply due to "line go up" being stupidly effective (compute getting cheaper, more training data being available, better optimization strategies, better architectures) [0]

Specifically this idea of small fine tuned LoRA models falls into the trap quite often: People assume you can beat the big, slow, general purpose LLMs with a small highly specialized model that has been fine tuned on the "good" human intuition of your special inhouse dataset.

LoRA can do great things, but it is often misunderstood what LoRA actually does.

0: http://www.incompleteideas.net/IncIdeas/BitterLesson.html

Re: LoRA Speedrun – a public wall-clock leaderboard for fine-tuning techniques

#30

Post by maleldil »

Schlagbohrer wrote: post #17
gowthamgts12 wrote: post #14

Earlier quoted context omitted.

Low Rank Adaptation. Google “Lora llm”

Yes but what are LoRAs being tested for / trying to do in this case? I couldn't find any information there about what the actual output goal is for the LoRAs. Speed doing what?

How long it takes to train that particular configuration. The table column is named "train time".
Post reply on HN