Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Mistral NeMo

mistral.ai

Post reply on HN
Print view View original

131–140 of 165 posts

Re: Mistral NeMo

#131

Post by chant4747 »

zone411 wrote: post #126

Interesting that the benchmarks they show have it outperforming Gemma 2 9B and Llama 3 8B, but it does a lot worse on my NYT Connections benchmark (5.1 vs 16.3 and 12.3). The new GPT-4o mini also does better at 14.3. It's just one benchmark though, so looking forward to additional scores.

Can you help me understand why people seem to think of Connections as a more robust indicator of (general) performance than benchmarks typically used for eval?

It seems to me that while the game is very challenging for people it’s not necessarily an indicator of generalization. I can see how it’s useful - but I have trouble seeing how a low score on it would indicate low performance on most tasks.

Thanks and hopefully this isn’t perceived as offensive. Just trying to learn more about it.

edit: I realize you yourself indicate that it's "just one benchmark" - I am more asking about the broader usage I have seen here on HN comments from several people.

Re: Mistral NeMo

#132

Post by troupo »

moffkalast wrote: post #116
troupo wrote: post #29

Earlier quoted context omitted.

The explanation is easier, I think. Consider what data these models are trained on, and who are the immediate developers of these models. The models are trained on a vast set of whatever is available on the internet. They are developed by tech people/programmers who are surprisingly blind to their own biases and interests. There's no surprise that one of the main things they want to try and do is programming, using v…

Then again, we just had this on the front page: https://news.ycombinator.com/item?id=40957990 > We first document a significant decline in stock trading volume during ChatGPT outages and find that the effect is stronger for firms with corporate news released immediately before or during the outages. We further document similar declines in the short-run price impact, return variance, and bid-ask spreads, consistent wi…

Stock trading is indistinguishable from gambling :)

But true, I forgot that this, too, is part of finance

Re: Mistral NeMo

#133

Post by xena »

LoganDark wrote: post #58

Is the base model unaligned? Disappointing to see alignment from allegedly "open" models.

The reason that companies align models is so that they don't get on the front page of the new york times with a headline like "Techaro's AI model used by terrorists to build a pipe bomb that destroyed the New York Stock Exchange datacentre".

Re: Mistral NeMo

#134

Post by troupo »

dannyw wrote: post #42
troupo wrote: post #29

Earlier quoted context omitted.

The explanation is easier, I think. Consider what data these models are trained on, and who are the immediate developers of these models. The models are trained on a vast set of whatever is available on the internet. They are developed by tech people/programmers who are surprisingly blind to their own biases and interests. There's no surprise that one of the main things they want to try and do is programming, using v…

> - be prepared to carry actual legal liability for the output of your models Section 230. It's been argued that a response by a LLM, to user input, is "user-generated content" and hence the platform has generally no liability (except CSAM). Nobody has successfully sued.

No one has challenged this. Because LLMs haven't been (widely) used in legal or legally binding contexts

Re: Mistral NeMo

#135

Post by imtringued »

wongarsu wrote: post #68
yjftsjthsd-h wrote: post #3

> Today, we are excited to release Mistral NeMo, a 12B model built in collaboration with NVIDIA. Mistral NeMo offers a large context window of up to 128k tokens. Its reasoning, world knowledge, and coding accuracy are state-of-the-art in its size category. As it relies on standard architecture, Mistral NeMo is easy to use and a drop-in replacement in any system using Mistral 7B. > We have released pre-trained base an…

You could consider the improvement in model performance a bit of a cheat - they beat other models "in the same size category" that have 30% fewer parameters. I still welcome this approach. 7B seems like a dead end in terms of reasoning and generalization. They are annoyingly close to statistical parrots, a world away from the moderate reasoning you get in 70B models. Any use case where that's useful can increasingly…

Except Llama 3 8b is a significant improvement over llama 2, which was basically so terrible that there was a whole community building fine tunes that are better than what the multi billion dollar company can do using a much smaller budget. With llama 3 8b things have shifted towards there being much less community fine-tunes that actually beat it. The fact that Mistral AI can still build models that beat it, means the company isn't falling too far behind a significantly better equipped competitor.

What's more irritating is that they decided to do quantization aware training for fp8. int8 quantization results in an imperceptible loss of quality that is difficult to pick up in benchmarks. They should have gone for something more aggressive like 4-bit, where quantization leads to a significant loss in quality.

Re: Mistral NeMo

#136

Post by imtringued »

xena wrote: post #12
yjftsjthsd-h wrote: post #3

> Today, we are excited to release Mistral NeMo, a 12B model built in collaboration with NVIDIA. Mistral NeMo offers a large context window of up to 128k tokens. Its reasoning, world knowledge, and coding accuracy are state-of-the-art in its size category. As it relies on standard architecture, Mistral NeMo is easy to use and a drop-in replacement in any system using Mistral 7B. > We have released pre-trained base an…

Easy head math: parameter count times parameter size plus 20-40% for inference slop space. Anywhere from 8-40GB of vram required depending on quantization levels being used.

They did quantization aware training for fp8 so you won't get any benefits from using more than 12GB of RAM for the parameters. What you might be using more RAM is the much bigger context window.

Re: Mistral NeMo

#137

Post by HanClinto »

HanClinto wrote: post #120
andrethegiant wrote: post #82

Earlier quoted context omitted.

Why would it take a couple days? Is it not a matter of uploading the model to their registry, or are there more steps involved than that?

Ollama depends on llama.cpp as its backend, so if there are any changes that need to be made to support anything new in this model architecture or tokenizer, then it will need to be added there first. Then the model needs to be properly quantized and formatted for GGUF (the model format that llama.cpp uses), tested, and uploaded to the model registry. So there's some length to the pipeline that things need to go thro…

Issue to track Mistral NeMo support in llama.cpp: https://github.com/ggerganov/llama.cpp/issues/8577

Re: Mistral NeMo

#138

Post by rustcleaner »

simpaticoder wrote: post #24
pixelatedindex wrote: post #14

Pardon me if this is a dumb question, but is it possible for me to download these models into my computer (I have a 1080ti and a [2|3]070ti) and generate some sort of api interface? That way I can write programs that calls this API, and I find this appealing. EDIT: This a 1W light bulb moment for me, thank you!

Justine Tunney (of redbean fame) is actively working on getting LLMs to run well on CPUs, where RAM is cheap. If successful this would eliminate an enormous bottleneck to running local models. If anyone can do this, she can. (And thank you to Mozilla for financially supporting her work). See https://justine.lol/matmul/ and https://github.com/mozilla-Ocho/llamafile

There's no good reason for consumer nvidia cards to lack SODIMM-like slots for video RAM, except to rake in big bucks and induce more hasty planned obsolescence.

Re: Mistral NeMo

#140

Post by regularfry »

candiddevmike wrote: post #84
gooob wrote: post #81

Earlier quoted context omitted.

no, not the actual content, just the titles of the content. like "book title" by "author". the tool just simply can't be taken seriously by anyone until they release that information. this is the case for all these models. it's ridiculous, almost insulting.

They can't release it without admitting to copyright infringement.

They can't do it without getting sued for copyright infringement. That's not quite the same.
Post reply on HN