Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Mixtral 8x22B

mistral.ai

Post reply on HN
Print view View original

151–160 of 252 posts

Re: Mixtral 8x22B

#151

Post by popf1 »

2Gkashmiri wrote: post #30
noman-land wrote: post #20

I'm really excited about this model. Just need someone to quantize it to ~3 bits so it'll run on a 64GB MacBook Pro. I've gotten a lot of use from the 8x7b model. Paired with llamafile and it's just so good.

Can you explain your use case? I tried to get into offline llms, on my machine and even android but without discrete graphics, its a slow hog so I didnt enjoy it but suppose I buy one, what then ?

> Can you explain your use case?

pretty sure you can run it un-censored... that would be my use case

Re: Mixtral 8x22B

#152

Post by popf1 »

sva_ wrote: post #99
kristopolous wrote: post #62

Earlier quoted context omitted.

Everything is soldered in these days. It's complete garbage. And most of the other vendors just copy Apple so even things like Lenovo have the same problems. The current state of laptops is such trash

Plenty of laptops still have SO-DIMM, such as EliteBook for example. People need to vote with their wallet, and not buy stuff that goes against their principles.

There are so many variables though ... most of the time you have to compromise on a few things.

Re: Mixtral 8x22B

#153

Post by azinman2 »

navbaker wrote: post #90
tinyhouse wrote: post #35

Earlier quoted context omitted.

Well, I don't have hardware to run a 141B parameters model, even if only 39B are active during inference.

It will be quantized in a matter of days and runnable on most laptops.

8 bit is 149G. 4 bit is 80G.

I wouldn’t call this runnable on most laptops.

Re: Mixtral 8x22B

#154

Post by Me1000 »

MacsHeadroom wrote: post #78
doublextremevil wrote: post #18

How much vram is need to run this?

80GB in 4bit. But because it only activates one expert at a time, it can run on a fast CPU in reasonable time. So 96GB of DDR4 will do. 96GB of DDR5 is better.

WizardLM-2 8x22b (which was a fine tune of the Mixtral 8x22b base model) at 4bit was only 80GB.

Re: Mixtral 8x22B

#155

Post by imjonse »

skissane wrote: post #101
imjonse wrote: post #15

Earlier quoted context omitted.

So far it is Command R+. Let's see how this will fare on Chatbot Arena after a few weeks of use.

> So far it is Command R+ Most people would not consider Command R+ to count as the "best permissively licensed model" since CC-BY-NC is not usually considered "permissively licensed" – the "NC" part means "non-commercial use only"

My bad, I remembered wrongly it was Apache too.

Re: Mixtral 8x22B

#157

Post by pseudosavant »

mdrzn wrote: post #14

"64K tokens context window" I do wish they had managed to extend it to at least 128K to match the capabilities of GPT-4 Turbo Maybe this limit will become a joke when looking back? Can you imagine reaching a trillion tokens context window in the future, as Sam speculated on Lex's podcast?

FWIW, the 128k context window for GPT-4 is only for input. I believe the output content is still only 4k.

Re: Mixtral 8x22B

#158

Post by GeekyBear »

sva_ wrote: post #99
kristopolous wrote: post #62

Earlier quoted context omitted.

Everything is soldered in these days. It's complete garbage. And most of the other vendors just copy Apple so even things like Lenovo have the same problems. The current state of laptops is such trash

Plenty of laptops still have SO-DIMM, such as EliteBook for example. People need to vote with their wallet, and not buy stuff that goes against their principles.

With SO-DIMM you gain expandability at the cost of higher power draw and latency as well as lower throughput.

> SO-DIMM memory is inherently slower than soldered memory. Moreover, considering the fact that SO-DIMM has a maximum speed of 6,400MHz means that it won’t be able to handle the DDR6 standard, which is already in the works.

https://fossbytes.com/camm2-ram-standard/

Re: Mixtral 8x22B

#159

Post by cjbprime »

woadwarrior01 wrote: post #124
byteknight wrote: post #113

First test I tried to run a random taxation question through it Output: https://gist.github.com/IAmStoxe/7fb224225ff13b1902b6d172467... Within the first paragraph, it outputs: > GET AN ESSAY WRITTEN FOR YOU FROM AS LOW AS $13/PAGE Thought that was hilarious.

Looks like an issue with the quantization that ollama (i.e llama.cpp) uses and not the model itself. It's common knowledge from Mixtral 8x7B that quantizing the MoE gates is pernicious to model perplexity. And yet they continue to do it. :)

No, it's unrelated to quantization, they just weren't using the instruct model.

Re: Mixtral 8x22B

#160

Post by cjbprime »

rgbrgb wrote: post #77
fire_lake wrote: post #54

Earlier quoted context omitted.

Why do we expect this to perform better? Couldn’t a regular network converge on this structure anyways?

Here's my naive intuition: in general bigger models can store more knowledge but take longer to do inference. MoE provides a way to blend the advantages of having a bigger model (more storage) with the advantages of having smaller models at inference time (faster, less memory required). When you do inference, tokens hit a small layer that is load balancing the experts then activate 1 or 2 experts. So you're storing r…

Not quite, you don't save memory, only compute.
Post reply on HN