Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Brave Leo now uses Mixtral 8x7B as default

brave.com

Post reply on HN
Print view View original

131–140 of 184 posts

Re: Brave Leo now uses Mixtral 8x7B as default

#131

Post by attentive »

jodleif wrote: post #3
rhdunn wrote: post #2

If you want to run Mixtral 8x7B locally you can use llama.cpp (including with any of the supporting libraries/interfaces such as text-generation-webui) with https://huggingface.co/TheBloke/Nous-Hermes-2-Mixtral-8x7B-S... . The smallest quantized version (2bit) needs 20GB of RAM (which can be offloaded onto the VRAM of a decent 4090 GPU). The 4bit quantized versions are the largest models that can just about fit onto…

I prefer koboldcpp over llama.cpp. It’s easy to spilt between gpu/cpu on models larger than VRAM

kobold bundles and runs llama.cpp. So it should be fairly the same with convenient defaults.

Re: Brave Leo now uses Mixtral 8x7B as default

#132

Post by kristianp »

FoodWThrow wrote: post #94
Diti wrote: post #85

Earlier quoted context omitted.

If by performance you mean browser performance, you have more performance with Firefox nowadays. https://news.ycombinator.com/item?id=36770883

I use firefox because it has the most hassle-free hardware decoding in linux. However, everything basically feels better with Brave, even with the same amount of plug-ins.

I found the snap update notifications too annoying on Ubuntu, so I tried the ppa. But it the video plugin would crash. So back to Chrome for me.

Re: Brave Leo now uses Mixtral 8x7B as default

#133

Post by jacooper »

jacooper wrote: post #61
mkesper wrote: post #57

Earlier quoted context omitted.

Or a Linux machine with a Ryzen using the internal GPU and the unified RAM (scroll down at llama.cpp and look for ROCm).

Wait ROCm support Ryzen APUs and still doesn't support dedicatedly GPUs like the 6700XT?!

Supports* dedicated*

Re: Brave Leo now uses Mixtral 8x7B as default

#134

Post by luke-stanley »

quinncom wrote: post #125
wolverine876 wrote: post #118

Kudos to Brave (for this and other privacy features): Unlinkable subscription: If you sign up for Leo Premium, you’re issued unlinkable tokens that validate your subscription when using Leo. This means that Brave can never connect your purchase details with your usage of the product, an extra step that ensures your activity is private to you and only you. The email you used to create your account is unlinkable to you…

This is very cool, and something I’d like to integrate in my own apps. Does anybody know how this works exactly, not using foreign keys?

I could guess, an "anonymous payment credential service" could do something like this: 1. User completes payment for the paid for service, 2. To track the payment entitlement, a random, unique ID is generated by the service for the user, that is not related to any of their data. 3. This ID is saved in a database as a valid payment key. 4. The database records IDs in shuffled batches, or with semi-random fuzzy / low resolution timestamps to prevent correlation between payment time and ID generation. 5. Each ID has an entitlement limit or usage stopping point, ensuring it's only valid for the subscribed period.

Another way might be Zero-Knowledge Proofs (ZKPs), but that might be more complex. They might even use their BAT crypto stuff for this somehow, I suppose. Whatever solution, would need a fundamental solution for how to avoid correlation, I think.

Re: Brave Leo now uses Mixtral 8x7B as default

#135

Post by Const-me »

beck5 wrote: post #27
rhdunn wrote: post #2

If you want to run Mixtral 8x7B locally you can use llama.cpp (including with any of the supporting libraries/interfaces such as text-generation-webui) with https://huggingface.co/TheBloke/Nous-Hermes-2-Mixtral-8x7B-S... . The smallest quantized version (2bit) needs 20GB of RAM (which can be offloaded onto the VRAM of a decent 4090 GPU). The 4bit quantized versions are the largest models that can just about fit onto…

Dumb question, but how can a 32 bit number be converted to 2 bits and still be useful? It seems like magic.

Here’s an example of a custom 4 bits/weight codec for ML weights:

https://github.com/Const-me/Cgml/blob/master/Readme.md#bcml1...

llama.cpp does it slightly differently but still, AFAIK their quantized data formats are conceptually similar to my codec.

Re: Brave Leo now uses Mixtral 8x7B as default

#136

Post by dimask »

baq wrote: post #77
sp332 wrote: post #34

Earlier quoted context omitted.

You wouldn’t be able to fit the whole model into 8GB VRAM. It’s faster than not using a GPU at all, but most of it would still be computed on the CPU.

IME ollama ran mixtral on a 1070 fast enough.

Though it most probably does not run in on the 1070 but rather on the cpu. It cannot fit on a 1070, it is not about speed, a 1070 cannot run it period.

Re: Brave Leo now uses Mixtral 8x7B as default

#137

Post by dimask »

mkesper wrote: post #57
tarruda wrote: post #13

Earlier quoted context omitted.

> You would need multiple GPUs with shared memory if you wanted to offload the higher precision models to VRAM. Or just a powerful apple silicon machine? I've tried dolphin mixtral 4bit on a 36gb ram MacBook m3, and inference is super fast.

Or a Linux machine with a Ryzen using the internal GPU and the unified RAM (scroll down at llama.cpp and look for ROCm).

How much RAM are you able to set aside for a ryzen igpu?

Re: Brave Leo now uses Mixtral 8x7B as default

#138

Post by trissi1996 »

jacooper wrote: post #133
jacooper wrote: post #61

Earlier quoted context omitted.

Wait ROCm support Ryzen APUs and still doesn't support dedicatedly GPUs like the 6700XT?!

Supports* dedicated*

While not being officially supported, rocm runs just fine on my 6700XT, i just have to set an env var(export HSA_OVERRIDE_GFX_VERSION=10.3.0)

Re: Brave Leo now uses Mixtral 8x7B as default

#139

Post by zaat »

dimask wrote: post #137
mkesper wrote: post #57

Earlier quoted context omitted.

Or a Linux machine with a Ryzen using the internal GPU and the unified RAM (scroll down at llama.cpp and look for ROCm).

How much RAM are you able to set aside for a ryzen igpu?

I think my motherboard allow me to dedicate 12. I didn't see any improvement using CPU + ROCm compared to CPU alone. Using CPU alone I can get 4.2 - 5 Tokens/s, with ROCm I can get 4.5 - 5.2 T/s. With CPU + RTX 2070 8GB I get 6.2-7 T/s.

Re: Brave Leo now uses Mixtral 8x7B as default

#140

Post by danShumway »

davikr wrote: post #4

It's nice using Brave because you have Chromium's better performance, without having to worry about Manifest V2 dying and taking adblocking down with it. I have uBlock Origin enabled, but it has barely caught anything that slipped past the browser filters.

This has come up a few times, but as far as I know Gorhill is planning to full-on drop support for Chromium browsers outside of uBO Lite. Does Brave keeping Manifest V2 matter if developers stop maintaining the Chromium version of those extensions?

I also still can't really find if Brave has an extension store or if it's connecting to Chrome's. If it's the latter, then it seems like V2 extensions are going away for Brave regardless of what API decisions it makes, because short of sideloading them the addons won't be on the Chrome web store anymore and won't be getting updates. I assume Brave supports sideloading extensions, and maybe developers would maintain extensions for Chromium that can't actually be used in Chrome? Although that seems a little optimistic. But not having a way to search for V2 extensions or get ratings/reviews seems like it will still be a problem for users.

Post reply on HN