31–40 of 255 posts
Some companies spend weeks on landing pages, demos and cute thought through promo videos and then there is Mistral, casually dropping a magnet link on Friday.
No public statement from Mistral yet. What we know: - Mixture of Experts architecture. - 8x 7B parameters experts (potentially trained starting with their base 7B model?). - 96GB of weights. You won't be able to run this on your home GPU.
People are running models 2-4 times that size on local GPUs.
What's more, this will run on a MacBook CPU just fine-- and at an extremely high speed.
Looks to be Mixture of Experts, here is the params.json: { "dim": 4096, "n_layers": 32, "head_dim": 128, "hidden_dim": 14336, "n_heads": 32, "n_kv_heads": 8, "norm_eps": 1e-05, "vocab_size": 32000, "moe": { "num_experts_per_tok": 2, "num_experts": 8 } }
Still 7B, but now with 32k context. Looking forward to see how it compares with the previous one, and what the community does with it.
It will run with the speed of a 7B model while being much smarter but requiring ~24GB of RAM instead of ~4GB (in 4bit).
Honest question: Why isn't this on Huggingface? Is this one a leaked model with a questionable training or alignment methodology? EDIT: I mean, I guess they didn't hack their own twitter account, but still.
It'll be on Huggingface soon. This is how they dropped their original 7B model as well. It's a marketing thing, but it works!
Earlier quoted context omitted.
Damn, I was hoping it was still a single 7B model that I would be able to run on my GPU
You can, wait for a 4-bit quantized version
Some companies spend weeks on landing pages, demos and cute thought through promo videos and then there is Mistral, casually dropping a magnet link on Friday.
I'm curious about their business model.
[1] https://www.bloomberg.com/news/articles/2023-12-04/openai-ri...
Looks to be Mixture of Experts, here is the params.json: { "dim": 4096, "n_layers": 32, "head_dim": 128, "hidden_dim": 14336, "n_heads": 32, "n_kv_heads": 8, "norm_eps": 1e-05, "vocab_size": 32000, "moe": { "num_experts_per_tok": 2, "num_experts": 8 } }
I don't see any code in there. What runtime could load these weights?
Everything open source is llama now. Facebook all but standardized the architecture.
I dunno about the moe. Is there existing transformers code for that part? It kinda looks like there is based on the config.