I still don’t understand the business model of releasing open source gen AI models. If this took 3072 H100s to train, why are they releasing it for free? I understand they charge people when renting from their platform, but why permit people to run it themselves?
> but why permit people to run it themselves?
I wouldn't worry about that if I were them: it's been shown again and again that people will pay for convenience.
What I'd worry about is Amazon/Cloudflare repackaging my model and outcompeting my platform.
I wonder why Mistral et al don't prepare GGUF versions of these for launch day? If I were them I'd want to be the default source of the versions of my models that people use, rather than farming that out to whichever third party races to publish the GGUF (and other formats) first.
Same could be said for onnx. Depends on which community you are in as to what you want.
Right - imagine how much of an impact a model release could have if it included GGUF and ONNX and MLX along with PyTorch.
I doubt they could. Their corpus almost certainly is mostly composed of copyrighted material they don't have a license for. It's an open question whether that's an issue for using it for model training, but it's obvious they wouldn't be allowed to distribute it as a corpus. That'd just be regular copyright infringement. Maybe they could share a list of the content of their corpus. But that wouldn't be too helpful and…
no, not the actual content, just the titles of the content. like "book title" by "author". the tool just simply can't be taken seriously by anyone until they release that information. this is the case for all these models. it's ridiculous, almost insulting.
Uh..
That would almost be worse. All copyright holders would need to do is search a list of titles if I'm understanding your proposal correctly.
I still don’t understand the business model of releasing open source gen AI models. If this took 3072 H100s to train, why are they releasing it for free? I understand they charge people when renting from their platform, but why permit people to run it themselves?
> but why permit people to run it themselves? I wouldn't worry about that if I were them: it's been shown again and again that people will pay for convenience. What I'd worry about is Amazon/Cloudflare repackaging my model and outcompeting my platform.
> What I'd worry about is Amazon/Cloudflare repackaging my model and outcompeting my platform.
> Today, we are excited to release Mistral NeMo, a 12B model built in collaboration with NVIDIA. Mistral NeMo offers a large context window of up to 128k tokens. Its reasoning, world knowledge, and coding accuracy are state-of-the-art in its size category. As it relies on standard architecture, Mistral NeMo is easy to use and a drop-in replacement in any system using Mistral 7B. > We have released pre-trained base an…
You could consider the improvement in model performance a bit of a cheat - they beat other models "in the same size category" that have 30% fewer parameters. I still welcome this approach. 7B seems like a dead end in terms of reasoning and generalization. They are annoyingly close to statistical parrots, a world away from the moderate reasoning you get in 70B models. Any use case where that's useful can increasingly…
I actually meant execution speed from quantisation awareness - agreed that comparing against smaller models is a bit cheating.
Justine Tunney (of redbean fame) is actively working on getting LLMs to run well on CPUs, where RAM is cheap. If successful this would eliminate an enormous bottleneck to running local models. If anyone can do this, she can. (And thank you to Mozilla for financially supporting her work). See https://justine.lol/matmul/ and https://github.com/mozilla-Ocho/llamafile
I think it's mostly the memory bandwidth though that makes the GPUs so fast with LLMs. My card does about 1TB/s. CPU RAM won't come near that. I'm sure a lot of optimisations can be had but I think GPUs will still be significantly ahead. Macs are so good at it because Apple solder the memory on top of the SoC for a really wide and low latency connection.
This is a good and valid comment. It is difficult to predict the future, but I would be curious what the best case theoretical performance of an LLM on a typical x86 or ARM system with DDR4 or DDR5 RAM. My uneducated guess is that it can be very good, perhaps 50% the speed of a specialized GPU/RAM device. In practical terms, the CPU approach is required for very large contexts, up to as large as the lifetime of all interactions you have with your LLM.
What's the reason for measuring the model size in context window length and not GB? Also, are these small models OSS? Easier self hosting seems to be the main benefo for small models.
I suspect you might be confusing the numbers: 12B (which is the very first number they give) is not context length, it's parameter count. The reason to use parameter count is because final size in GB depends on quantization. A 12B model at 8 bit parameter width would be 12Gbytes (plus some % overhead), while at 16 bit would be 24Gbytes. Context length here is 128k which is orthogonal to model size. You can notice the…
> but why permit people to run it themselves? I wouldn't worry about that if I were them: it's been shown again and again that people will pay for convenience. What I'd worry about is Amazon/Cloudflare repackaging my model and outcompeting my platform.
> What I'd worry about is Amazon/Cloudflare repackaging my model and outcompeting my platform. Why let Amazon/Cloudflare repackage it?