Hi all, I'm the main maintainer from the OpenLLM team here. I'm actively developing the fine-tuning feature and will release a PR soon enough. Stay tuned. In the meanwhile, the best way to track the development workflow is at our discord, so feel free to join!!
OpenLLM
31–40 of 175 posts
Re: OpenLLM
#32Stray thought: It would be better to specify NNs in terms of their training-size to weight-size in bytes. Rather than "No. Parameters", or at least, this ratio with the number of parameters. So, eg., I'd imagine ChatGPT would be, say: 100s PB in 0.5TB. The number of parameters is a nearly meaningless metric, consider, eg., that if all the parameters covary then there's one "functional" parameter. The compression rati…
I’m not sure ML researchers would agree that number of (compressed) bytes are more meaningful than number of parameters. Parameters have mathematical meaning – bytes doesn’t.
My point is that they don't have a mathematical meaning. They have a training-time impact, but that's more relevant when creating than using.
You'd need to know how many parameters were independently covarying for any given class of predictions. It certainly isnt all of them.
You could cite the "average dropout percent to random-level accuracy on a given class of problems" (my guess is that this would show 5-20% of parameters could be dropped).
My point, I suppose, is that users of NNs arent interested in the architecture characteristics which affect training -- they're interested in how capable any given model will be.
For this we really want to know how large the training data was, and how compressed it has been. If it's 1PB -> 1MB then we can easily say that's much less useful (but much faster to use) than 1PB -> 0.5TB.
Likewise we can say, if it's a video generator, that 1PB is far too small to be generally useful -- so at best it'll be domain-speicifc.
Re: OpenLLM
#33Hi all, I'm the main maintainer from the OpenLLM team here. I'm actively developing the fine-tuning feature and will release a PR soon enough. Stay tuned. In the meanwhile, the best way to track the development workflow is at our discord, so feel free to join!!
Thanks for the great project! Any chance, your team might consider more open platform than Discord for posting updates? I personally find Discord hard to use, and there’s no way to have sensible subscription (like RSS). Discord is usually muted.
Related: As an operator/mod/admin it's fairly straight-forward to bridge a Discord channel to Matrix (and, if one so desires, from there to IRC), allowing users not on Discord to participate. Conservative mods concerned about spam can start with an allowlist for which servers can join.
Re: OpenLLM
#34Hi all, I'm the main maintainer from the OpenLLM team here. I'm actively developing the fine-tuning feature and will release a PR soon enough. Stay tuned. In the meanwhile, the best way to track the development workflow is at our discord, so feel free to join!!
How can we stay tuned if we can't do tuning? :P
Fine-tuning is coming up in the next release!
You can actually try it out on the main branch :P
Re: OpenLLM
#35This sounds promising. Smaller but custom trained/tuned models would be ideal - works for the task without the overhead
In my experience it doesn't work like that. It's like if you take an idiot and spend a bunch of time training them, they'll still perform much worse than a moderately intelligent person with much less training. And smaller models can be pretty idiotic.
Then why do Chess AI perform much better than LLMs trying to play chess.
Re: OpenLLM
#36Earlier quoted context omitted.
In my experience it doesn't work like that. It's like if you take an idiot and spend a bunch of time training them, they'll still perform much worse than a moderately intelligent person with much less training. And smaller models can be pretty idiotic.
Then why do Chess AI perform much better than LLMs trying to play chess.
Because they do completely different things? They literally have nothing to do with each other. Why do planes fly better than ships if ChatGPT can't do math?
Re: OpenLLM
#37Earlier quoted context omitted.
In my experience it doesn't work like that. It's like if you take an idiot and spend a bunch of time training them, they'll still perform much worse than a moderately intelligent person with much less training. And smaller models can be pretty idiotic.
Then why do Chess AI perform much better than LLMs trying to play chess.
Provided that the problem is suited to the strengths of an LLM at all. An example might be a small ai custom trained on documentation for libraries. You ask it a question like "how do I make the background move with parallax effect when you move the cursor". It's a little ambiguous, high-level concept, and probably not a single function.
Small ai: likely makes up a function or suggests a single function which isn't sufficient. Refuses to budge from its answer or apologies and gets confused
Large LLM: able to actually understand the question, combine several functions. If it doesn't work you can tell it why and it fixes it
Re: OpenLLM
#38This sounds promising. Smaller but custom trained/tuned models would be ideal - works for the task without the overhead
Agreed.
Re: OpenLLM
#39Do we know how LLMs available in OpenLLM and other open source LLMs compare to different versions of GPT models? I know there’s a leaderboard on huggingface: https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderb... but it doesn’t contain GPT models.
Re: OpenLLM
#40Earlier quoted context omitted.
In my experience it doesn't work like that. It's like if you take an idiot and spend a bunch of time training them, they'll still perform much worse than a moderately intelligent person with much less training. And smaller models can be pretty idiotic.
Then why do Chess AI perform much better than LLMs trying to play chess.
Why would a language model be good at playing chess?