What are good API providers that serve mixtral? I know only octo ai which seems decent but will be good to know alternatives too
Brave Leo now uses Mixtral 8x7B as default
11–20 of 184 posts
Re: Brave Leo now uses Mixtral 8x7B as default
#12If you want to run Mixtral 8x7B locally you can use llama.cpp (including with any of the supporting libraries/interfaces such as text-generation-webui) with https://huggingface.co/TheBloke/Nous-Hermes-2-Mixtral-8x7B-S... . The smallest quantized version (2bit) needs 20GB of RAM (which can be offloaded onto the VRAM of a decent 4090 GPU). The 4bit quantized versions are the largest models that can just about fit onto…
I prefer koboldcpp over llama.cpp. It’s easy to spilt between gpu/cpu on models larger than VRAM
Runs in Oobabooga textUi as well, if you add the llama.cpp extension. Easier interface imo, plus fun stuff like coqui and whisper integration.
Re: Brave Leo now uses Mixtral 8x7B as default
#13If you want to run Mixtral 8x7B locally you can use llama.cpp (including with any of the supporting libraries/interfaces such as text-generation-webui) with https://huggingface.co/TheBloke/Nous-Hermes-2-Mixtral-8x7B-S... . The smallest quantized version (2bit) needs 20GB of RAM (which can be offloaded onto the VRAM of a decent 4090 GPU). The 4bit quantized versions are the largest models that can just about fit onto…
> You would need multiple GPUs with shared memory if you wanted to offload the higher precision models to VRAM.
Or just a powerful apple silicon machine? I've tried dolphin mixtral 4bit on a 36gb ram MacBook m3, and inference is super fast.
Re: Brave Leo now uses Mixtral 8x7B as default
#14If you want to run Mixtral 8x7B locally you can use llama.cpp (including with any of the supporting libraries/interfaces such as text-generation-webui) with https://huggingface.co/TheBloke/Nous-Hermes-2-Mixtral-8x7B-S... . The smallest quantized version (2bit) needs 20GB of RAM (which can be offloaded onto the VRAM of a decent 4090 GPU). The 4bit quantized versions are the largest models that can just about fit onto…
I prefer koboldcpp over llama.cpp. It’s easy to spilt between gpu/cpu on models larger than VRAM
That's interesting. It also looks like koboldcpp works better with long interactions, as it only processes changed tokens. I'm using llama.cpp with text-generation-webui and its OpenAI compatible API. I'll have to look to see if I can use koboldcpp with it.
Re: Brave Leo now uses Mixtral 8x7B as default
#15If you want to run Mixtral 8x7B locally you can use llama.cpp (including with any of the supporting libraries/interfaces such as text-generation-webui) with https://huggingface.co/TheBloke/Nous-Hermes-2-Mixtral-8x7B-S... . The smallest quantized version (2bit) needs 20GB of RAM (which can be offloaded onto the VRAM of a decent 4090 GPU). The 4bit quantized versions are the largest models that can just about fit onto…
`ollama run mixtral:8x7b-instruct-v0.1-q3_K_L` works fast on my 3090 locally
Re: Brave Leo now uses Mixtral 8x7B as default
#16It's nice using Brave because you have Chromium's better performance, without having to worry about Manifest V2 dying and taking adblocking down with it. I have uBlock Origin enabled, but it has barely caught anything that slipped past the browser filters.
Brave"s support of Manifest V3 is totally dependent on Google and Chrome
>Brave will support uBO and uMatrix so long as Google doesn’t remove underlying V2 code paths (which seem to be needed for Chrome for enterprise support, so should stay in the Chromium open source)
Re: Brave Leo now uses Mixtral 8x7B as default
#17If you want to run Mixtral 8x7B locally you can use llama.cpp (including with any of the supporting libraries/interfaces such as text-generation-webui) with https://huggingface.co/TheBloke/Nous-Hermes-2-Mixtral-8x7B-S... . The smallest quantized version (2bit) needs 20GB of RAM (which can be offloaded onto the VRAM of a decent 4090 GPU). The 4bit quantized versions are the largest models that can just about fit onto…
Why not normal RAM? Ryzen 5600 with 128GB DDR4 is perfectly fine to run mixtral 8bit, and costs less than $1000. GPUs are only needed if you can not wait 5 minutes for an answer, or for training.
Why not both? Llama.cpp allows layering GGUF models between GPU and CPU memory.
Re: Brave Leo now uses Mixtral 8x7B as default
#18It's nice using Brave because you have Chromium's better performance, without having to worry about Manifest V2 dying and taking adblocking down with it. I have uBlock Origin enabled, but it has barely caught anything that slipped past the browser filters.
Brave"s support of Manifest V3 is totally dependent on Google and Chrome >Brave will support uBO and uMatrix so long as Google doesn’t remove underlying V2 code paths (which seem to be needed for Chrome for enterprise support, so should stay in the Chromium open source) https://twitter.com/BrendanEich/status/1534893414579249152
Yeah, but the Brave adblocker is built-in, it's not an extension.
Re: Brave Leo now uses Mixtral 8x7B as default
#19It's nice using Brave because you have Chromium's better performance, without having to worry about Manifest V2 dying and taking adblocking down with it. I have uBlock Origin enabled, but it has barely caught anything that slipped past the browser filters.
Brave"s support of Manifest V3 is totally dependent on Google and Chrome >Brave will support uBO and uMatrix so long as Google doesn’t remove underlying V2 code paths (which seem to be needed for Chrome for enterprise support, so should stay in the Chromium open source) https://twitter.com/BrendanEich/status/1534893414579249152
It doesn't really matter much in practice. The built-in adblocker (which does not rely on extension APIs) has excellent performance, about as effective as ublock origin, and supports the same filter lists.
Re: Brave Leo now uses Mixtral 8x7B as default
#20Earlier quoted context omitted.
Brave"s support of Manifest V3 is totally dependent on Google and Chrome >Brave will support uBO and uMatrix so long as Google doesn’t remove underlying V2 code paths (which seem to be needed for Chrome for enterprise support, so should stay in the Chromium open source) https://twitter.com/BrendanEich/status/1534893414579249152
It doesn't really matter much in practice. The built-in adblocker (which does not rely on extension APIs) has excellent performance, about as effective as ublock origin, and supports the same filter lists.
It does matter if you use other extensions that require Manifest V2