Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Mistral NeMo

mistral.ai

Post reply on HN
Print view View original

61–70 of 165 posts

Re: Mistral NeMo

#61

Post by bernaferrari »

yjftsjthsd-h wrote: post #3

> Today, we are excited to release Mistral NeMo, a 12B model built in collaboration with NVIDIA. Mistral NeMo offers a large context window of up to 128k tokens. Its reasoning, world knowledge, and coding accuracy are state-of-the-art in its size category. As it relies on standard architecture, Mistral NeMo is easy to use and a drop-in replacement in any system using Mistral 7B. > We have released pre-trained base an…

if you want to be lazy, 7b = 7gb of vRAM, 12b = 12gb of vRAM, but quantizing you might be able to do with with ~6-8. So any 16gb Macbook could run it (but not much else).

Re: Mistral NeMo

#62

Post by simion314 »

k__ wrote: post #46

What's the reason for measuring the model size in context window length and not GB? Also, are these small models OSS? Easier self hosting seems to be the main benefo for small models.

>What's the reason for measuring the model size in context window length and not GB?

there are 2 different things.

The context window is how many tokens ii's context can contain, so on a big model you could put in the context a few books and articles and then start your questions, on a small context model you can start a conversation and after a short time it will start forgetting eh first prompts. Big context will use more memory and will cost on performance but imagine you could give it your entire code project and then you can ask it questions, so often I know there is some functions already there that does soemthing but I can't remember the name.

Re: Mistral NeMo

#63

Post by benmanns »

dannyw wrote: post #47
jorgesborges wrote: post #18

I’m AI stupid. Does anyone know if training on multiple languages provides “cross-over” — so training done in German can be utilized when answering a prompt in English? I once went through various Wikipedia articles in a couple languages and the differences were interesting. For some reason I thought they’d be almost verbatim (forgetting that’s not how Wikipedia works!) and while I can’t remember exactly I felt they…

Anecdata, but I did some continued pretraining on a toy LLM using machine-translated data; of the original dataset. Performance improved across all benchmarks; in English (the original language).

Am I understanding correctly? You look an English dataset, trained an LLM, machine translated the English dataset to e.g. Spanish, continued training the model, and performance for queries in English improved? That’s really interesting.

Re: Mistral NeMo

#64

Post by madeofpalk »

miki123211 wrote: post #52
madeofpalk wrote: post #20

I find it interesting how coding/software development still appears to be the one category that these most popular models release specialised models for. Where's the finance or legal models from Mistral or Meta or OpenAI? Perhaps it's just confirmation bias, but programming really does seem to be the ideal usecase for LLMs in a way that other professions just haven't been able to crack. Compared to other types of wor…

> Where's the finance or legal models from Mistral or Meta or OpenAI? Programming is "weird" in that it requires both specialized knowledge and specialized languages, and the languages are very different from any language that humans speak. Legal requires specialized knowledge, but legal writing is still just English and it follows English grammar rules, although it's sometimes a very strange "dialect" of English. Fi…

Yes, it appears to be the clear successful usecase for the technology, in a way that hasn't been replicated for other professions.

I remain very sceptical that a chat-like interface is the ideal form for LLMs, yet it seems very optimal for programming specifically, along with Copilot-like interfaces of just outputting text.

Re: Mistral NeMo

#65

Post by sofixa »

Raed667 wrote: post #37
mythz wrote: post #10

Earlier quoted context omitted.

IMO Google's Gemma2 27B [1] is the sweet spot for running locally on commodity 16GB VRAM cards. [1] https://ollama.com/library/gemma2:27b

If I "only" have 16GB of ram on a macbook pro, would that still work ?

If it's an M-series one with "unified memory" (shared RAM between the CPU, GPU and NPU on the same chip), yes.

Re: Mistral NeMo

#66

Post by Bumblonono »

yjftsjthsd-h wrote: post #3

> Today, we are excited to release Mistral NeMo, a 12B model built in collaboration with NVIDIA. Mistral NeMo offers a large context window of up to 128k tokens. Its reasoning, world knowledge, and coding accuracy are state-of-the-art in its size category. As it relies on standard architecture, Mistral NeMo is easy to use and a drop-in replacement in any system using Mistral 7B. > We have released pre-trained base an…

It fits a 4090. Nvidia lists the models and therefore i assume 24gig is min

Re: Mistral NeMo

#67

Post by hislaziness »

bernaferrari wrote: post #61
yjftsjthsd-h wrote: post #3

> Today, we are excited to release Mistral NeMo, a 12B model built in collaboration with NVIDIA. Mistral NeMo offers a large context window of up to 128k tokens. Its reasoning, world knowledge, and coding accuracy are state-of-the-art in its size category. As it relies on standard architecture, Mistral NeMo is easy to use and a drop-in replacement in any system using Mistral 7B. > We have released pre-trained base an…

if you want to be lazy, 7b = 7gb of vRAM, 12b = 12gb of vRAM, but quantizing you might be able to do with with ~6-8. So any 16gb Macbook could run it (but not much else).

isn't it 2 bytes (fp16) per param. so 7b = 14 GB+some for inference?

Re: Mistral NeMo

#68

Post by wongarsu »

yjftsjthsd-h wrote: post #3

> Today, we are excited to release Mistral NeMo, a 12B model built in collaboration with NVIDIA. Mistral NeMo offers a large context window of up to 128k tokens. Its reasoning, world knowledge, and coding accuracy are state-of-the-art in its size category. As it relies on standard architecture, Mistral NeMo is easy to use and a drop-in replacement in any system using Mistral 7B. > We have released pre-trained base an…

You could consider the improvement in model performance a bit of a cheat - they beat other models "in the same size category" that have 30% fewer parameters.

I still welcome this approach. 7B seems like a dead end in terms of reasoning and generalization. They are annoyingly close to statistical parrots, a world away from the moderate reasoning you get in 70B models. Any use case where that's useful can increasingly be filled by even smaller models, so chasing slightly larger models to get a bit more "intelligence" might be the right move

Re: Mistral NeMo

#69

Post by madeofpalk »

sofixa wrote: post #60
madeofpalk wrote: post #20

I find it interesting how coding/software development still appears to be the one category that these most popular models release specialised models for. Where's the finance or legal models from Mistral or Meta or OpenAI? Perhaps it's just confirmation bias, but programming really does seem to be the ideal usecase for LLMs in a way that other professions just haven't been able to crack. Compared to other types of wor…

Finance already has their own models and has had them for decades. Market predictions and high frequency trading is literally what all the hedge funds and the like have been doing for a few decades now. Including advanced sources of information like (take with a grain of salt, I've heard it on the internet) using satellite images to measure factory activity and thus predict results. Understandably they're all quite s…

I guess finance is not in need of a large language model?

Re: Mistral NeMo

#70

Post by planb »

miki123211 wrote: post #31
jorgesborges wrote: post #18

I’m AI stupid. Does anyone know if training on multiple languages provides “cross-over” — so training done in German can be utilized when answering a prompt in English? I once went through various Wikipedia articles in a couple languages and the differences were interesting. For some reason I thought they’d be almost verbatim (forgetting that’s not how Wikipedia works!) and while I can’t remember exactly I felt they…

Generally yes, with caveats. There was some research showing that training a model on facts like "the mother of John Smith is Alice" but in German allowed it to answer questions like "who's the mother of John Smith", but not questions like "what's the name of Alice's child", regardless of language. Not sure if this holds at larger model sizes though, it's the sort of problem that's usually fixable by throwing more pa…

> Language models definitely do generalize to some extend and they're not "stochastic parrots" as previously thought, but there are some weird ways in which we expect them to generalize but they don't.

Do you have any good sources that explain this? I was always thinking LLMs are indeed stochastic parrots, but language (that is the unified corpus of all languages in the training data) already inherently contains the „generalization“. So the intelligence is encoded in the language humans speak.

Post reply on HN