Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Brave Leo now uses Mixtral 8x7B as default

brave.com

Post reply on HN
Print view View original

31–40 of 184 posts

Re: Brave Leo now uses Mixtral 8x7B as default

#31

Post by pennaMan »

tarruda wrote: post #13
rhdunn wrote: post #2

If you want to run Mixtral 8x7B locally you can use llama.cpp (including with any of the supporting libraries/interfaces such as text-generation-webui) with https://huggingface.co/TheBloke/Nous-Hermes-2-Mixtral-8x7B-S... . The smallest quantized version (2bit) needs 20GB of RAM (which can be offloaded onto the VRAM of a decent 4090 GPU). The 4bit quantized versions are the largest models that can just about fit onto…

> You would need multiple GPUs with shared memory if you wanted to offload the higher precision models to VRAM. Or just a powerful apple silicon machine? I've tried dolphin mixtral 4bit on a 36gb ram MacBook m3, and inference is super fast.

I can run 4bit on a beat up 1070 ti. GP talks about higher precision models

Re: Brave Leo now uses Mixtral 8x7B as default

#32

Post by charcircuit »

davikr wrote: post #4

It's nice using Brave because you have Chromium's better performance, without having to worry about Manifest V2 dying and taking adblocking down with it. I have uBlock Origin enabled, but it has barely caught anything that slipped past the browser filters.

MV3 doesn't prevent adblockers from existing.

Re: Brave Leo now uses Mixtral 8x7B as default

#34

Post by sp332 »

pennaMan wrote: post #31
tarruda wrote: post #13

Earlier quoted context omitted.

> You would need multiple GPUs with shared memory if you wanted to offload the higher precision models to VRAM. Or just a powerful apple silicon machine? I've tried dolphin mixtral 4bit on a 36gb ram MacBook m3, and inference is super fast.

I can run 4bit on a beat up 1070 ti. GP talks about higher precision models

You wouldn’t be able to fit the whole model into 8GB VRAM. It’s faster than not using a GPU at all, but most of it would still be computed on the CPU.

Re: Brave Leo now uses Mixtral 8x7B as default

#35

Post by sp332 »

beck5 wrote: post #27
rhdunn wrote: post #2

If you want to run Mixtral 8x7B locally you can use llama.cpp (including with any of the supporting libraries/interfaces such as text-generation-webui) with https://huggingface.co/TheBloke/Nous-Hermes-2-Mixtral-8x7B-S... . The smallest quantized version (2bit) needs 20GB of RAM (which can be offloaded onto the VRAM of a decent 4090 GPU). The 4bit quantized versions are the largest models that can just about fit onto…

Dumb question, but how can a 32 bit number be converted to 2 bits and still be useful? It seems like magic.

The extra precision is more useful for training. Once the network is optimized, it's a statistical model and only needs enough precision to make good guesses. In fact, one of the big papers on this also pointed out that you can drop about 40% of the weights completely. I think people generally skip that part because sparse matrix operations are slower, so it doesn’t help here.

Re: Brave Leo now uses Mixtral 8x7B as default

#36

Post by Gracana »

MPSimmons wrote: post #25
snowfield wrote: post #9

Earlier quoted context omitted.

Or if you want multiple sessions at the same time. Or if you want to do anything else with your machine while it's running. But realistically, 5 minutes is too long. It should be conversational, and for that you need at least 5 tokens per second. Which your Ryzen just can't do.

>It should be conversational, and for that you need at least 5 tokens per second. To be fair, a lot of people are using this for non-interactive work, like batching document analysis or offline processing of user generated content.

In my experience, it takes some experimentation to figure out a good prompt. I don’t think I would have gotten very far off I had to wait that long for each result.

Re: Brave Leo now uses Mixtral 8x7B as default

#37

Post by sp332 »

jodleif wrote: post #3
rhdunn wrote: post #2

If you want to run Mixtral 8x7B locally you can use llama.cpp (including with any of the supporting libraries/interfaces such as text-generation-webui) with https://huggingface.co/TheBloke/Nous-Hermes-2-Mixtral-8x7B-S... . The smallest quantized version (2bit) needs 20GB of RAM (which can be offloaded onto the VRAM of a decent 4090 GPU). The 4bit quantized versions are the largest models that can just about fit onto…

I prefer koboldcpp over llama.cpp. It’s easy to spilt between gpu/cpu on models larger than VRAM

Llama.cpp has --n-gpu-layers that lets you set how much of the model to put on the GPU.

Re: Brave Leo now uses Mixtral 8x7B as default

#38

Post by beefield »

beck5 wrote: post #27
rhdunn wrote: post #2

If you want to run Mixtral 8x7B locally you can use llama.cpp (including with any of the supporting libraries/interfaces such as text-generation-webui) with https://huggingface.co/TheBloke/Nous-Hermes-2-Mixtral-8x7B-S... . The smallest quantized version (2bit) needs 20GB of RAM (which can be offloaded onto the VRAM of a decent 4090 GPU). The 4bit quantized versions are the largest models that can just about fit onto…

Dumb question, but how can a 32 bit number be converted to 2 bits and still be useful? It seems like magic.

I think of it with this kind of analogy: the original image is stored with 32 bit color scheme. You can reduce the color scheme to 16 bit accuracy and still figure out pretty well what the image is about. 2 bit is stretching this to a bit far, basically either pixel is white or it is black, but even if you lose lots of nuances in the image, in many images even that gives you some idea whats going on in the image.

Re: Brave Leo now uses Mixtral 8x7B as default

#40

Post by mcbuilder »

thebruce87m wrote: post #30
deely3 wrote: post #8

Earlier quoted context omitted.

Is this submarine comment?

What is the definition of a submarine comment? Google fails and ChatGPT says: > A "submarine comment" on social media refers to a comment that is made on an old post or thread, long after the conversation has died down. This term derives from the idea of a submarine which remains submerged and out of sight for long periods before suddenly surfacing. In the context of social media, it's when someone delves deep into s…

I think GP is trying to coin a term for stealth marketing Hacker News comments, except the analogy doesn't really make sense.
Post reply on HN