Asked Mistral 8x7B for an essay on ham. It started telling me about Hamlet.
It must start from the beginning. Pig > piglet. Ham > Hamlet
Brave Leo now uses Mixtral 8x7B as default
81–90 of 184 posts
Re: Brave Leo now uses Mixtral 8x7B as default
#82Interesting, I must have missed the first Leo announcement. I really like how privacy conscious it is. They don’t store any chat record which is what I want.
There is no way to confirm that claim, just like there is no way to confirm that a VPN service is "no log".
You gotta trust them by their word
Re: Brave Leo now uses Mixtral 8x7B as default
#83Interesting, I must have missed the first Leo announcement. I really like how privacy conscious it is. They don’t store any chat record which is what I want.
There is no way to confirm that claim, just like there is no way to confirm that a VPN service is "no log".
Yes, at some point if you're going to interface with other humans you will eventually just have to trust their word.
For some people's threat models that isn't good enough, but for the vast majority of people—people who aren't being pursued by state intelligence agencies but who are squeamish about how much data a company like Google collects—a pinky promise from Brave or Mullvad is good enough.
Re: Brave Leo now uses Mixtral 8x7B as default
#84If you want to run Mixtral 8x7B locally you can use llama.cpp (including with any of the supporting libraries/interfaces such as text-generation-webui) with https://huggingface.co/TheBloke/Nous-Hermes-2-Mixtral-8x7B-S... . The smallest quantized version (2bit) needs 20GB of RAM (which can be offloaded onto the VRAM of a decent 4090 GPU). The 4bit quantized versions are the largest models that can just about fit onto…
2bit is pretty damn terrible, I don't recommend it for anything serious.
Re: Brave Leo now uses Mixtral 8x7B as default
#85It's nice using Brave because you have Chromium's better performance, without having to worry about Manifest V2 dying and taking adblocking down with it. I have uBlock Origin enabled, but it has barely caught anything that slipped past the browser filters.
If by performance you mean browser performance, you have more performance with Firefox nowadays. https://news.ycombinator.com/item?id=36770883
Re: Brave Leo now uses Mixtral 8x7B as default
#86What are good API providers that serve mixtral? I know only octo ai which seems decent but will be good to know alternatives too
Together.ai seems to be the best, incredibly fast.
Re: Brave Leo now uses Mixtral 8x7B as default
#87If you want to run Mixtral 8x7B locally you can use llama.cpp (including with any of the supporting libraries/interfaces such as text-generation-webui) with https://huggingface.co/TheBloke/Nous-Hermes-2-Mixtral-8x7B-S... . The smallest quantized version (2bit) needs 20GB of RAM (which can be offloaded onto the VRAM of a decent 4090 GPU). The 4bit quantized versions are the largest models that can just about fit onto…
Dumb question, but how can a 32 bit number be converted to 2 bits and still be useful? It seems like magic.
This blog post might shed some light on the matter. If I'm understanding it correctly, it claims there are emergent features on the LLM weights that make it easier to "compress" the floats into smaller bits without losing much precision.
https://timdettmers.com/2022/08/17/llm-int8-and-emergent-fea...
Note that 2 bit quantization is generally regarded as too aggressive. Generally 4bits+ achieves a good tradeoff, see eg. https://arxiv.org/abs/2212.09720
Re: Brave Leo now uses Mixtral 8x7B as default
#88It's nice using Brave because you have Chromium's better performance, without having to worry about Manifest V2 dying and taking adblocking down with it. I have uBlock Origin enabled, but it has barely caught anything that slipped past the browser filters.
If by performance you mean browser performance, you have more performance with Firefox nowadays. https://news.ycombinator.com/item?id=36770883
While still not being as secure as chromium and still not supporting many advanced features like WebGpu.
Re: Brave Leo now uses Mixtral 8x7B as default
#89If you want to run Mixtral 8x7B locally you can use llama.cpp (including with any of the supporting libraries/interfaces such as text-generation-webui) with https://huggingface.co/TheBloke/Nous-Hermes-2-Mixtral-8x7B-S... . The smallest quantized version (2bit) needs 20GB of RAM (which can be offloaded onto the VRAM of a decent 4090 GPU). The 4bit quantized versions are the largest models that can just about fit onto…
Why not normal RAM? Ryzen 5600 with 128GB DDR4 is perfectly fine to run mixtral 8bit, and costs less than $1000. GPUs are only needed if you can not wait 5 minutes for an answer, or for training.
Not sure why you're getting downvoted. It performs decent enough on my Ryzen 3600X with 64GB of RAM. It definitely wouldn't be usable for production or fine-tuning, but it's fine for experimenting.
Re: Brave Leo now uses Mixtral 8x7B as default
#90What are good API providers that serve mixtral? I know only octo ai which seems decent but will be good to know alternatives too
The creators of the model actually have their own platform where you can access this model and others via API: https://console.mistral.ai/