Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Mistral NeMo

mistral.ai

Post reply on HN
Print view View original

71–80 of 165 posts

Re: Mistral NeMo

#71

Post by dpflan »

These big models are getting pumped out like crazy, that is the business of these companies. But basically, it feels like private/industry just figured out how to scale up a scalable process (deep learning), and it required not $M research grants but $BB "research grants"/funding, and the scaling laws seem to be fun to play with and tweak more interesting things out of these and find cool "emergent" behavior as billions of data points get correlated.

But pumping out models and putting artifacts on HuggingFace, is that a business? What are these models being used for? There is a new one at a decent clip.

Re: Mistral NeMo

#72

Post by 317070 »

madeofpalk wrote: post #20

I find it interesting how coding/software development still appears to be the one category that these most popular models release specialised models for. Where's the finance or legal models from Mistral or Meta or OpenAI? Perhaps it's just confirmation bias, but programming really does seem to be the ideal usecase for LLMs in a way that other professions just haven't been able to crack. Compared to other types of wor…

I work in the field. The reason has not been mentioned yet.

It's because (for an unknown reason), having coding and software development in the training mix is really helpful at most other tasks. It improves everything to do with logical thinking by a large margin, and that seems to help with many other downstream tasks.

Even if you don't need the programming, you want it in the training mix to get that logical thinking, which is hard to get from other resources.

I don't know how much that is true for legal or financial resources.

Re: Mistral NeMo

#73

Post by ancientworldnow »

hislaziness wrote: post #67
bernaferrari wrote: post #61

Earlier quoted context omitted.

if you want to be lazy, 7b = 7gb of vRAM, 12b = 12gb of vRAM, but quantizing you might be able to do with with ~6-8. So any 16gb Macbook could run it (but not much else).

isn't it 2 bytes (fp16) per param. so 7b = 14 GB+some for inference?

This was trained to be run at FP8 with no quality loss.

Re: Mistral NeMo

#75

Post by kaoD »

k__ wrote: post #46

What's the reason for measuring the model size in context window length and not GB? Also, are these small models OSS? Easier self hosting seems to be the main benefo for small models.

I suspect you might be confusing the numbers: 12B (which is the very first number they give) is not context length, it's parameter count.

The reason to use parameter count is because final size in GB depends on quantization. A 12B model at 8 bit parameter width would be 12Gbytes (plus some % overhead), while at 16 bit would be 24Gbytes.

Context length here is 128k which is orthogonal to model size. You can notice the specify both parameters and context size because you need both to characterize an LLM.

It's also interesting to know what parameter width it was trained on because you cannot get more information by "quantizing wider" -- it only makes sense to quantize into a narrower parameter width to save space.

Re: Mistral NeMo

#76

Post by Palmik »

voiper1 wrote: post #11
Workaccount2 wrote: post #5

Is "Parameter Creep" going to becomes a thing? They hold up Llama-8b as a competitor despite NeMo having 50% more parameters. The same thing happened with gemma-27b, where they compared it to all the 7-9b models. It seems like an easy way to boost benchmarks while coming off as "small" at first glance.

Oddly, they are only charging slightly more for their hosted version: open-mistral-7b is 25c/m tokens open-mistral-nemo-2407 is 30c/m tokens https://mistral.ai/technology/#pricing

They specifically call out fp8 aware training and TensoRT LLM is really good (efficient) with fp8 inference on H100 and other hopper cards. It's possible that they run the 7b natively in fp16 as smaller models suffer more from even "modest" quantization like this.

Re: Mistral NeMo

#77

Post by drewmate »

madeofpalk wrote: post #20

I find it interesting how coding/software development still appears to be the one category that these most popular models release specialised models for. Where's the finance or legal models from Mistral or Meta or OpenAI? Perhaps it's just confirmation bias, but programming really does seem to be the ideal usecase for LLMs in a way that other professions just haven't been able to crack. Compared to other types of wor…

It's just easier to iterate and improve on a coding specialist AI when that is also the skill required to iterate on said AI.

Products that build on general LLM tech are already being used in other fields. For example, my lawyer friend has started using one by LexisNexis[0] and is duly impressed by how it works. It's only a matter of time before models like that get increasingly specialized for that kind of work, it's just harder for lawyers to drive that kind of change alone. Plus, there's a lot more resistance in 'legacy' professions to any kind of change, much less one that is perceived to threaten the livelihoods of established professionals.

Current LLMs are already not bad at a lot of things, but lawyer bots, accountant bots and more are likely coming.

[0] https://www.lexisnexis.com/en-us/products/lexis-plus-ai.page

Re: Mistral NeMo

#78

Post by Foobar8568 »

madeofpalk wrote: post #69
sofixa wrote: post #60

Earlier quoted context omitted.

Finance already has their own models and has had them for decades. Market predictions and high frequency trading is literally what all the hedge funds and the like have been doing for a few decades now. Including advanced sources of information like (take with a grain of salt, I've heard it on the internet) using satellite images to measure factory activity and thus predict results. Understandably they're all quite s…

I guess finance is not in need of a large language model?

It does but everything is a joke...

Re: Mistral NeMo

#79

Post by ofermend »

Congrats. Very exciting to see continued innovation around smaller models, that can perform much better than larger models. This enables faster inference and makes them more ubiquitous.

Re: Mistral NeMo

#80

Post by mistercheph »

wongarsu wrote: post #68
yjftsjthsd-h wrote: post #3

> Today, we are excited to release Mistral NeMo, a 12B model built in collaboration with NVIDIA. Mistral NeMo offers a large context window of up to 128k tokens. Its reasoning, world knowledge, and coding accuracy are state-of-the-art in its size category. As it relies on standard architecture, Mistral NeMo is easy to use and a drop-in replacement in any system using Mistral 7B. > We have released pre-trained base an…

You could consider the improvement in model performance a bit of a cheat - they beat other models "in the same size category" that have 30% fewer parameters. I still welcome this approach. 7B seems like a dead end in terms of reasoning and generalization. They are annoyingly close to statistical parrots, a world away from the moderate reasoning you get in 70B models. Any use case where that's useful can increasingly…

I strongly disagree, have you used fp16 or q8 llama3 8b?
Post reply on HN