I think there is real value in going smaller/limiting resources. The trend is 'just make the weights bigger and throw more data at it'. It is a MBA's view of winning. We have a knob, keep turning it. It does work but it may not drive as much creativity as resource limits can drive. It is like urban growth boundaries in city planning. If you aren't allowed to 'just expand' you are forced to build more intelligently inside the city and those creative solutions often lead to major improvements.
LoRA Speedrun – a public wall-clock leaderboard for fine-tuning techniques
11–20 of 39 posts
Re: LoRA Speedrun – a public wall-clock leaderboard for fine-tuning techniques
#12Feels like i'm missing the introduction paragraph in the repo.
What is LoRA in this context? the communication protocol? Or another term appropriated by LLMs? Why the speedrun?
Re: LoRA Speedrun – a public wall-clock leaderboard for fine-tuning techniques
#13Sadly 100% generated. I think the idea is interesting though, although I wonder if training time for LoRA is such a bottleneck to deserve its own, extremely narrowly scoped, leaderboard. Maybe if it was more tasks or more models we could hope that it transfers? With a single task, and a single model, I’d be afraid of this overfitting pretty heavily. For NanoGPT, I think the idea always was that the ideas can be trans…
That's fair on both points. Much of this was built with AI, but the runs and numbers are real. They are also reproducible, so I would prefer to be judged on that. And yes, using a single model and task can lead to overfitting. The plan is to add more tracks, including bigger models and other tasks, so a technique only matters if it transfers. Right now, it's just the initial track, so your concern is valid. Thanks fo…
AI generates or not - irrelevant if the substance is high quality. Always good to see projects.
You should consider hosting this leaderboard on hugging face, you will get a lot more AI enthusiasts and practitioners there.
Re: LoRA Speedrun – a public wall-clock leaderboard for fine-tuning techniques
#14Feels like i'm missing the introduction paragraph in the repo. What is LoRA in this context? the communication protocol? Or another term appropriated by LLMs? Why the speedrun?
Low Rank Adaptation. Google “Lora llm”
Re: LoRA Speedrun – a public wall-clock leaderboard for fine-tuning techniques
#15[flagged]
Re: LoRA Speedrun – a public wall-clock leaderboard for fine-tuning techniques
#16I think there is real value in going smaller/limiting resources. The trend is 'just make the weights bigger and throw more data at it'. It is a MBA's view of winning. We have a knob, keep turning it. It does work but it may not drive as much creativity as resource limits can drive. It is like urban growth boundaries in city planning. If you aren't allowed to 'just expand' you are forced to build more intelligently in…
You are right only in so far that it is more economical.
But it is not the MBA's view of winning, it's just one potential conclusion you could draw from the bitter lesson of Machine Learning. As long as the need for more intelligence outpaces the economics of using intelligence, you'll get bigger models. This idea that small, fine-tuned models can outperform bigger models capabilities wise is mostly misinformed. They are genuinely good at other metrics, but sadly more actually means better in ML-land (most of the time at least).
Re: LoRA Speedrun – a public wall-clock leaderboard for fine-tuning techniques
#17Feels like i'm missing the introduction paragraph in the repo. What is LoRA in this context? the communication protocol? Or another term appropriated by LLMs? Why the speedrun?
Low Rank Adaptation. Google “Lora llm”
Yes but what are LoRAs being tested for / trying to do in this case? I couldn't find any information there about what the actual output goal is for the LoRAs. Speed doing what?
Re: LoRA Speedrun – a public wall-clock leaderboard for fine-tuning techniques
#18I think there is real value in going smaller/limiting resources. The trend is 'just make the weights bigger and throw more data at it'. It is a MBA's view of winning. We have a knob, keep turning it. It does work but it may not drive as much creativity as resource limits can drive. It is like urban growth boundaries in city planning. If you aren't allowed to 'just expand' you are forced to build more intelligently in…
> It is a MBA's view of winning
A deeply ironic comment which associates with due to complete ignorance about the group. An MBA would never approve a technique with basically unlimited capex. So I hate to break it to you but "bigger weights" is 100% the computer scientist's view of winning because everything is an "abstraction".
Re: LoRA Speedrun – a public wall-clock leaderboard for fine-tuning techniques
#19Sadly 100% generated. I think the idea is interesting though, although I wonder if training time for LoRA is such a bottleneck to deserve its own, extremely narrowly scoped, leaderboard. Maybe if it was more tasks or more models we could hope that it transfers? With a single task, and a single model, I’d be afraid of this overfitting pretty heavily. For NanoGPT, I think the idea always was that the ideas can be trans…
I understand the feedback in the second paragraph, however I do not understand why we're judging projects by whether they've been AI generated or not. Have we stopped treating software as a black box? This behavior will only lead to devs moving away from OSS to avoid the AI stigma.
> Have we stopped treating software as a black box?
When did we begin to? Why?
Re: LoRA Speedrun – a public wall-clock leaderboard for fine-tuning techniques
#20[flagged]