Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Mistral "Mixtral" 8x7B 32k model [magnet]

twitter.com

Post reply on HN
Print view View original

171–180 of 255 posts

Re: Mistral "Mixtral" 8x7B 32k model [magnet]

#171

Post by puttycat »

stavros wrote: post #88
brucethemoose2 wrote: post #57

In other llm news, Mistral/Yi finetunes trained with a new (still undocumented) technique called "neural alignment" are blasting other models in the HF leaderboard. The 7B is "beating" most 70Bs. The 34B in testing seems... Very good: https://huggingface.co/fblgit/una-xaberius-34b-v1beta https://huggingface.co/fblgit/una-cybertron-7b-v2-bf16 I mention this because it could theoretically be applied to Mistral Moe. If…

Aren't LLM benchmarks at best irrelevant, at worst lying, at this point?

I wonder how it will rank on benchmarks which are password-protected to prevent test contamination, for example: https://github.com/taucompling/bliss

Re: Mistral "Mixtral" 8x7B 32k model [magnet]

#172

Post by valine »

fblgit wrote: post #166
valine wrote: post #160

Earlier quoted context omitted.

How much data do you need for UNA? Is a typical fine tuning dataset needed or can you get away with less than that?

doesn't require much data, in a 7B can take a couple hours ~

That’s cool. A couple hours on a single GPU or like 8x a100s?

Re: Mistral "Mixtral" 8x7B 32k model [magnet]

#173

Post by rishabhjain1198 »

moffkalast wrote: post #69
sockaddr wrote: post #66

Earlier quoted context omitted.

What does expert mean in this context?

It means it's 8 7B models in a trench coat in a sense, it runs as fast as a 14B (2 experts at a time apparently) but takes up as much memory as a 40B model (70% * 8 * 7B). There is some process trained into it that chooses which experts to use based on the question posed. GPT 4 is allegedly based on the same architecture, but at 8*222B.

In a MoE model with experts_per_token = 2 and each expert having 7B params, after picking the experts it should run as fast as the slowest 7B expert, not a comparable 14B model.

Re: Mistral "Mixtral" 8x7B 32k model [magnet]

#174

Post by rishabhjain1198 »

tavavex wrote: post #140
moffkalast wrote: post #69

Earlier quoted context omitted.

It means it's 8 7B models in a trench coat in a sense, it runs as fast as a 14B (2 experts at a time apparently) but takes up as much memory as a 40B model (70% * 8 * 7B). There is some process trained into it that chooses which experts to use based on the question posed. GPT 4 is allegedly based on the same architecture, but at 8*222B.

Does anyone here know roughly how an expert gets chosen? It seems like a very open-ended problem, and I'm not sure on how it can be implemented easily.

[Relevant paper](https://arxiv.org/abs/1701.06538).

TL;DR you can think of it as the initial part of the model is essentially dedicated to learning which experts to choose.

Re: Mistral "Mixtral" 8x7B 32k model [magnet]

#175

Post by 3abiton »

brucethemoose2 wrote: post #57

In other llm news, Mistral/Yi finetunes trained with a new (still undocumented) technique called "neural alignment" are blasting other models in the HF leaderboard. The 7B is "beating" most 70Bs. The 34B in testing seems... Very good: https://huggingface.co/fblgit/una-xaberius-34b-v1beta https://huggingface.co/fblgit/una-cybertron-7b-v2-bf16 I mention this because it could theoretically be applied to Mistral Moe. If…

HF leaderboards are rarely reflective of real world performance especially in small variations, but nonetheless, this is promising. What are the HW requirements for this latest Mistral7B?

Re: Mistral "Mixtral" 8x7B 32k model [magnet]

#176

Post by eurekin »

brucethemoose2 wrote: post #57

In other llm news, Mistral/Yi finetunes trained with a new (still undocumented) technique called "neural alignment" are blasting other models in the HF leaderboard. The 7B is "beating" most 70Bs. The 34B in testing seems... Very good: https://huggingface.co/fblgit/una-xaberius-34b-v1beta https://huggingface.co/fblgit/una-cybertron-7b-v2-bf16 I mention this because it could theoretically be applied to Mistral Moe. If…

I just played with 7b version. It really feels different than anything I tried before. It could explain a docker compose file. It generated a simple vue application component.

I asked around a bit about the example and it was strangely coherent and focused across the whole conversation. It was really well detecting, where I'm starting a new thread (without clearing a context) or referring to things before.

It caught me off guard as well with this:

> me: What does following mean [content of the docker compose]

> cybertron-7b: In the provided YAML configuration, "following" refers to specifying dependencies

I've never seen any model using my exact wording in quotes in conversation like that.

Re: Mistral "Mixtral" 8x7B 32k model [magnet]

#177

Post by refulgentis »

CamperBob2 wrote: post #110
coolspot wrote: post #99

Earlier quoted context omitted.

> $4500 Which is more than a price of RTX A6000 48gb ($4k used on ebay)

How fast does it run on that?

quantization makes it hard to have exactly one answer -- I'd make a q0 joke, except that's real now -- i.e. reduce the 3.4 * 10^38 range of float 32 to 2, a boolean.

it's not very good, at all, but now we can claim some pretty massive speedups.

I can't find anything for llama 2 70B on 4090 after 10 minutes of poking around, 13B is about 30 tkn/s. it looks like people generally don't run 70B unless they have multiple 4090s.

Re: Mistral "Mixtral" 8x7B 32k model [magnet]

#178

Post by wadefletch »

oh_sigh wrote: post #143
peanuty1 wrote: post #118

Earlier quoted context omitted.

Google Bard is still not available in Canada.

Are there some regulatory reasons why it would not be available? It seems weird if Google would intentionally block users merely to block them.

There's a proposed framework[1] in the EU that's rather restrictive. Seems like they're just not even bothering, perhaps to make a point.

[1] https://digital-strategy.ec.europa.eu/en/policies/regulatory...

Re: Mistral "Mixtral" 8x7B 32k model [magnet]

#179

Post by [deleted] »

moffkalast wrote: post #69
sockaddr wrote: post #66

Earlier quoted context omitted.

What does expert mean in this context?

It means it's 8 7B models in a trench coat in a sense, it runs as fast as a 14B (2 experts at a time apparently) but takes up as much memory as a 40B model (70% * 8 * 7B). There is some process trained into it that chooses which experts to use based on the question posed. GPT 4 is allegedly based on the same architecture, but at 8*222B.

[deleted]

Re: Mistral "Mixtral" 8x7B 32k model [magnet]

#180

Post by ComputerGuru »

brucethemoose2 wrote: post #95
_boffin_ wrote: post #87

Earlier quoted context omitted.

Interesting. One thing i noticed is that Mistral has a `max_position_embeddings` of ~32k while these have it at 4096. Any thoughts on that?

Is complicated. The 7B model (cybertron) is trained on Mistral. Mistral is technically a 32K model, but it uses a sliding window beyond 32K, and for all practical purposes in current implementations it behaves like an 8K model. The 34B model is based on Yi 34B, which is inexplicably marked as a 4K model in the config but actually works out to 32K if you literally just edit that line. Yi also has a 200K base model...…

Did you mean "but it uses a sliding window beyond" *8K*? Because I don't understand how the sentence would work otherwise.
Post reply on HN