Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Mistral NeMo

mistral.ai

Post reply on HN
Print view View original

81–90 of 165 posts

Re: Mistral NeMo

#81

Post by gooob »

wongarsu wrote: post #54
mcemilg wrote: post #34

I believe that if Mistral is serious about advancing in open source, they should consider sharing the corpus used for training their models, at least the base models pretraining data.

I doubt they could. Their corpus almost certainly is mostly composed of copyrighted material they don't have a license for. It's an open question whether that's an issue for using it for model training, but it's obvious they wouldn't be allowed to distribute it as a corpus. That'd just be regular copyright infringement. Maybe they could share a list of the content of their corpus. But that wouldn't be too helpful and…

no, not the actual content, just the titles of the content. like "book title" by "author". the tool just simply can't be taken seriously by anyone until they release that information. this is the case for all these models. it's ridiculous, almost insulting.

Re: Mistral NeMo

#82

Post by andrethegiant »

codetrotter wrote: post #21
pixelatedindex wrote: post #14

Pardon me if this is a dumb question, but is it possible for me to download these models into my computer (I have a 1080ti and a [2|3]070ti) and generate some sort of api interface? That way I can write programs that calls this API, and I find this appealing. EDIT: This a 1W light bulb moment for me, thank you!

I’d probably check https://ollama.com/library?q=Nemo in a couple of days. My guess is that by then ollama will have support for it. And you can then run the model locally on your machine with ollama.

Why would it take a couple days? Is it not a matter of uploading the model to their registry, or are there more steps involved than that?

Re: Mistral NeMo

#84

Post by candiddevmike »

gooob wrote: post #81
wongarsu wrote: post #54

Earlier quoted context omitted.

I doubt they could. Their corpus almost certainly is mostly composed of copyrighted material they don't have a license for. It's an open question whether that's an issue for using it for model training, but it's obvious they wouldn't be allowed to distribute it as a corpus. That'd just be regular copyright infringement. Maybe they could share a list of the content of their corpus. But that wouldn't be too helpful and…

no, not the actual content, just the titles of the content. like "book title" by "author". the tool just simply can't be taken seriously by anyone until they release that information. this is the case for all these models. it's ridiculous, almost insulting.

They can't release it without admitting to copyright infringement.

Re: Mistral NeMo

#85

Post by michaelt »

Bumblonono wrote: post #66
yjftsjthsd-h wrote: post #3

> Today, we are excited to release Mistral NeMo, a 12B model built in collaboration with NVIDIA. Mistral NeMo offers a large context window of up to 128k tokens. Its reasoning, world knowledge, and coding accuracy are state-of-the-art in its size category. As it relies on standard architecture, Mistral NeMo is easy to use and a drop-in replacement in any system using Mistral 7B. > We have released pre-trained base an…

It fits a 4090. Nvidia lists the models and therefore i assume 24gig is min

A 4090 will just narrowly fit a 34B parameter model at 4-bit quantisation.

A 12B model will run on a 4090 with plenty room to spare, even with 8-bit quantisation.

Re: Mistral NeMo

#86

Post by wkat4242 »

simpaticoder wrote: post #24
pixelatedindex wrote: post #14

Pardon me if this is a dumb question, but is it possible for me to download these models into my computer (I have a 1080ti and a [2|3]070ti) and generate some sort of api interface? That way I can write programs that calls this API, and I find this appealing. EDIT: This a 1W light bulb moment for me, thank you!

Justine Tunney (of redbean fame) is actively working on getting LLMs to run well on CPUs, where RAM is cheap. If successful this would eliminate an enormous bottleneck to running local models. If anyone can do this, she can. (And thank you to Mozilla for financially supporting her work). See https://justine.lol/matmul/ and https://github.com/mozilla-Ocho/llamafile

I think it's mostly the memory bandwidth though that makes the GPUs so fast with LLMs. My card does about 1TB/s. CPU RAM won't come near that. I'm sure a lot of optimisations can be had but I think GPUs will still be significantly ahead.

Macs are so good at it because Apple solder the memory on top of the SoC for a really wide and low latency connection.

Re: Mistral NeMo

#87

Post by d13 »

pixelatedindex wrote: post #14

Pardon me if this is a dumb question, but is it possible for me to download these models into my computer (I have a 1080ti and a [2|3]070ti) and generate some sort of api interface? That way I can write programs that calls this API, and I find this appealing. EDIT: This a 1W light bulb moment for me, thank you!

Try Lm Studio or Ollama. Load up the model, and there you go.

Re: Mistral NeMo

#88

Post by michaelt »

planb wrote: post #70
miki123211 wrote: post #31

Earlier quoted context omitted.

Generally yes, with caveats. There was some research showing that training a model on facts like "the mother of John Smith is Alice" but in German allowed it to answer questions like "who's the mother of John Smith", but not questions like "what's the name of Alice's child", regardless of language. Not sure if this holds at larger model sizes though, it's the sort of problem that's usually fixable by throwing more pa…

> Language models definitely do generalize to some extend and they're not "stochastic parrots" as previously thought, but there are some weird ways in which we expect them to generalize but they don't. Do you have any good sources that explain this? I was always thinking LLMs are indeed stochastic parrots, but language (that is the unified corpus of all languages in the training data) already inherently contains the…

I don't have explanations but I can point you to one of the papers: https://arxiv.org/pdf/2309.12288 which calls it "the reversal curse" and does a bunch of experiments showing models that are successful at questions like "Who is Tom Cruise’s mother?" (Mary Lee Pfeiffer) will not be equally successful at answering "Who is Mary Lee Pfeiffer’s son?"

Re: Mistral NeMo

#89

Post by andrethegiant »

I still don’t understand the business model of releasing open source gen AI models. If this took 3072 H100s to train, why are they releasing it for free? I understand they charge people when renting from their platform, but why permit people to run it themselves?

Re: Mistral NeMo

#90

Post by amrrs »

wongarsu wrote: post #68
yjftsjthsd-h wrote: post #3

> Today, we are excited to release Mistral NeMo, a 12B model built in collaboration with NVIDIA. Mistral NeMo offers a large context window of up to 128k tokens. Its reasoning, world knowledge, and coding accuracy are state-of-the-art in its size category. As it relies on standard architecture, Mistral NeMo is easy to use and a drop-in replacement in any system using Mistral 7B. > We have released pre-trained base an…

You could consider the improvement in model performance a bit of a cheat - they beat other models "in the same size category" that have 30% fewer parameters. I still welcome this approach. 7B seems like a dead end in terms of reasoning and generalization. They are annoyingly close to statistical parrots, a world away from the moderate reasoning you get in 70B models. Any use case where that's useful can increasingly…

>reasoning and generalization

any example use-cases or prompts? how do you define those?

Post reply on HN