21–30 of 83 posts
There's a British comedy skit lurking in here. "So it's a small large language model?" "Oh yes, very small." "How can it be small and large at the same time?" "Well, it's small by the standards of a large language model." "So it's large." "Oh yes, very large." "Large compared to what?" "Small language models." "And so something like ChatGPT, what would that be exactly? A large large language model?" "Yes, precisely.…
There's a British comedy skit lurking in here. "So it's a small large language model?" "Oh yes, very small." "How can it be small and large at the same time?" "Well, it's small by the standards of a large language model." "So it's large." "Oh yes, very large." "Large compared to what?" "Small language models." "And so something like ChatGPT, what would that be exactly? A large large language model?" "Yes, precisely.…
There's a British comedy skit lurking in here. "So it's a small large language model?" "Oh yes, very small." "How can it be small and large at the same time?" "Well, it's small by the standards of a large language model." "So it's large." "Oh yes, very large." "Large compared to what?" "Small language models." "And so something like ChatGPT, what would that be exactly? A large large language model?" "Yes, precisely.…
Mostly SOTA performance at the 3B level. A notable addition to the small but truly open club of models that provide full disclosure, code, recipes to reproduce their work. Looks like ballpark a million dollars of GPU time if you want to train up one for yourself (4000 gpus/24 days). Very nice write up that’s generous in sharing their learnings. This is a solid and positive contribution.
It's 384 H100s for 24 days, costing less than half a million dollars.
Like if I really really just wanted to build it from scratch, could I do so? (not that I have that money but just curious)
Earlier quoted context omitted.
It's 384 H100s for 24 days, costing less than half a million dollars.
Pardon me, but is the dataset public. Like if I really really just wanted to build it from scratch, could I do so? (not that I have that money but just curious)
Earlier quoted context omitted.
Pardon me, but is the dataset public. Like if I really really just wanted to build it from scratch, could I do so? (not that I have that money but just curious)
yes, both core web datasets are publicly available as well as the rest
To be honest, if I might argue then that this is one of the best truly open source models that we have got.
There is AllenAI and (Elmo?) and there is also this one which does distributed training but I think this looks a lot like SOTA for 3B parameters to me.
Thanks for telling me, I am not going to lie, I am going to try to test it now! (Ima try some GGUF since ollama convenience)
Which small model is good for fine tuning to various enterprise data sets? Our business units are wanting to run small models in browser and on mobile devices, without dealing with RAG and cloud resources.
You really need to try them all out yourself and make sure you have proper benchmarks. While machine learning is not my field, I've tried to finetune Mistral 7B (following their official guide and toolset) and the results did not satisfy. Had a few very specific questions from the dataset that no matter how much I've finetuned and tweaked the process it was not able to respond with correct information. A mix of vecto…
There's a British comedy skit lurking in here. "So it's a small large language model?" "Oh yes, very small." "How can it be small and large at the same time?" "Well, it's small by the standards of a large language model." "So it's large." "Oh yes, very large." "Large compared to what?" "Small language models." "And so something like ChatGPT, what would that be exactly? A large large language model?" "Yes, precisely.…