It wasn't clear but how much hardware does it take to run Mixtral 8x22B (mistral.ai) next to me locally?
Mixtral 8x22B
61–70 of 252 posts
Re: Mixtral 8x22B
#62These LLMs are making RAM great again. Wish I had invested in the extra 32GB for my mac laptop.
You can't upgrade it? Edit: I haven't owned a laptop for years, probably could have surmised they'd be more user hostile nowadays.
Everything is soldered in these days.
It's complete garbage. And most of the other vendors just copy Apple so even things like Lenovo have the same problems.
The current state of laptops is such trash
Re: Mixtral 8x22B
#63I'm really excited about this model. Just need someone to quantize it to ~3 bits so it'll run on a 64GB MacBook Pro. I've gotten a lot of use from the 8x7b model. Paired with llamafile and it's just so good.
Can you explain your use case? I tried to get into offline llms, on my machine and even android but without discrete graphics, its a slow hog so I didnt enjoy it but suppose I buy one, what then ?
I run Mistral-7B on an old laptop. It's not very fast and it's not very good, but it's just good enough to be useful.
My use case is that I'm more productive working with a LLM but being online is a constant temptation and distraction.
Most of the time I'll reach for offline docs to verify. So the LLM just points me in the right direction.
I also miss Google offline, so I'm working on a search engine. I thought I could skip crawling by just downloading common crawl, but unfortnately it's enormous and mostly junk or unsuitable for my needs. So my next project is how to data-mine common crawl to extract just the interesting (to me) bits...
When I have a search engine and a LLM I'll be able to run my own Phind, which will be really cool.
Re: Mixtral 8x22B
#64Re: Mixtral 8x22B
#65These LLMs are making RAM great again. Wish I had invested in the extra 32GB for my mac laptop.
You can't upgrade it? Edit: I haven't owned a laptop for years, probably could have surmised they'd be more user hostile nowadays.
I really really like my Macbook Pro. But dammit, you can't upgrade the thing (Mac laptops aren't upgrade-able anymore). I got M1 Max in 2021 with 32GB of RAM. I did not anticipate needing more than 32GB for anything I'd be doing on it. Turns out, a couple of years later, I like to run local LLMs that max out my available memory.
Re: Mixtral 8x22B
#66Earlier quoted context omitted.
This is a bit of a misnomer. Each expert is a sub network that specializes in sub understanding we can't possibly track. During training a routing network is punished if it does not evenly distribute training tokens to the correct experts. This prevents any one or two networks from becoming the primary networks. The result of this is that each token has essentially even probability of being routed to one of the sub m…
Why do we expect this to perform better? Couldn’t a regular network converge on this structure anyways?
[deleted]
Re: Mixtral 8x22B
#67Earlier quoted context omitted.
You can't upgrade it? Edit: I haven't owned a laptop for years, probably could have surmised they'd be more user hostile nowadays.
I really really like my Macbook Pro. But dammit, you can't upgrade the thing (Mac laptops aren't upgrade-able anymore). I got M1 Max in 2021 with 32GB of RAM. I did not anticipate needing more than 32GB for anything I'd be doing on it. Turns out, a couple of years later, I like to run local LLMs that max out my available memory.
I say 2021, but truth is the supply chain was so trash that year that it took almost a year to actually get delivered. I don't think I actually started using the thing until 2022.
Re: Mixtral 8x22B
#68"64K tokens context window" I do wish they had managed to extend it to at least 128K to match the capabilities of GPT-4 Turbo Maybe this limit will become a joke when looking back? Can you imagine reaching a trillion tokens context window in the future, as Sam speculated on Lex's podcast?
maybe we'll look back at token context windows like we look back at how much ram we have in a system.
I still don’t have enough RAM though ?
Re: Mixtral 8x22B
#69It feels absolutely amazing to build an AI startup right now. It's as if your product automatically becomes cheaper, more reliable, and more scalable with each new major model release. - We first struggled with limited context windows [solved] - We had issues with consistent JSON ouput [solved] - We had rate limiting and performance issues for the large 3rd party models [solved] - Hosting our own OSS models for small…
We are going to quickly reach the point where most of these AI startups (which do nothing but provide thin wrappers on top of public LLMs) aren't going to be needed at all. The differentiation will need to come from the value of the end product put in front of customers, not the AI backend.
Sure, in the same way SaaS companies are just thin wrappers on top of databases and the open web.
Re: Mixtral 8x22B
#70Earlier quoted context omitted.
This is a bit of a misnomer. Each expert is a sub network that specializes in sub understanding we can't possibly track. During training a routing network is punished if it does not evenly distribute training tokens to the correct experts. This prevents any one or two networks from becoming the primary networks. The result of this is that each token has essentially even probability of being routed to one of the sub m…
Why do we expect this to perform better? Couldn’t a regular network converge on this structure anyways?
It is a type of ensemble model. A regular network could do it, but a MoE will select a subset to do the task faster than the whole model would.