Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Brave Leo now uses Mixtral 8x7B as default

brave.com

Post reply on HN
Print view View original

151–160 of 184 posts

Re: Brave Leo now uses Mixtral 8x7B as default

#151

Post by attentive »

tacoman wrote: post #95
zaat wrote: post #51

Earlier quoted context omitted.

Using TheBloke nous-hermes-2-mixtral-8x7b-dpo.Q5_K_M.gguf, llam.cpp with oobabooga, on Ryzen 9 7950x and RTX 2070 super, offloading 4 layers to the GPU. 27,631MiB system memory used. 7358/8192MiB usage on the GPU. I'm getting 6.32 tokens/second. You: You are Hunter S. Thompson, write a 450 words essay on the LHC project. The writing must be in Thompson's style AI: Well, buckle up, kiddos, because we're about to dive…

I am using the exact same model. Ryzen 5600G w/32GB and an Nvidia P40 w/24GB VRAM 20/33 layers offloaded to GPU, 4K context. Uses 25GB system RAM and all 24GB VRAM. 5-7 tokens per second.

and Groq does 485.08 T/s on mixtral 8x7B-32k

I am not sure local models have any future other than POC/research. Depends on the cost of course.

Re: Brave Leo now uses Mixtral 8x7B as default

#152

Post by attentive »

firtoz wrote: post #11

What are good API providers that serve mixtral? I know only octo ai which seems decent but will be good to know alternatives too

I just discovered Groq, which does 485.08 T/s on mixtral 8x7B-32k

No idea on pricing but supposedly one can email to api@groq.com

Re: Brave Leo now uses Mixtral 8x7B as default

#153

Post by charcircuit »

danShumway wrote: post #149
charcircuit wrote: post #144

Earlier quoted context omitted.

>That is not what we mean by dynamic filters. My point was that "reduces ad blockers to static filter lists" is not true as it is not a static list of rules. >What critics of Manifest V3 are talking about is not the ability to dynamically add rules Yes, they are. Due to early versions of declarativeNetRequest only having a static list of rules there are a large amount of people who believe this to still be true. On t…

> My point was that "reduces ad blockers to static filter lists" is not true as it is not a static list of rules. That's not what we mean by static either. What we mean by static is that extensions lose the ability to dynamically analyze requests and block them using on-the-fly logic. > Yes, they are. Due to early versions of declarativeNetRequest only having a static list of rules there are a large amount of people…

>That's not what we mean by static either.

What is a static filter list other than a list of filters that does not change?

>When you replied to gkbrk saying that dynamic filters were supported you were not correcting misinformation

I was saying that the filter list was dynamic and not static.

>"Just inject scripts into the webpage" is a wild take

I was pointing out the upper bounds of what was possible and not giving recommendation on how it should be done.

>So it's not supported.

That is right.

>you want me to pretend that developer concerns have been addressed

No, I want people interested in this feature to track the bug or contribute the fix themselves since it is an open source project.

>it is reasonable to conclude that either Google does not see addon support as a priority worth investing sufficient developer resources into

That is correct in my understanding. This means that the open source community needs to dedicate their own resources if they want to accelerate progress here.

>the inability of an extension to load before the browser starts sending requests is a non-issue to you

Assuming no events are dropped then I don't see it as an issue.

>doesn't make the wild decision to start loading pages before the user's addons have initialized.

It doesn't sound that wild if it speeds up launch time by not blocking on third party extensions having to load.

>it's also about when it happens and the fact that (as mentioned above) uBO Lite can't guarantee that it will be running when a request is made

This is not an issue as the extension does not need to be running for the browser to apply the rules. The extension is only needed to update the rules.

>Well, it's a good thing that Google is waiting for MV3 to become mature and it's a good thing that MV2 won't stop working in June, right? /s

Considering that the deadline keeps being pushed back, I think they recognize that it has low maturity.

Re: Brave Leo now uses Mixtral 8x7B as default

#154

Post by tarruda »

asimpleusecase wrote: post #64
tarruda wrote: post #13

Earlier quoted context omitted.

> You would need multiple GPUs with shared memory if you wanted to offload the higher precision models to VRAM. Or just a powerful apple silicon machine? I've tried dolphin mixtral 4bit on a 36gb ram MacBook m3, and inference is super fast.

Did you do anything special to make that work? Is it useful? Or just a toy?

Nothing special other than llama.cpp, which is an inference engine optimized for apple silicon.

I heard you can simply install ollama app which uses llama.cpp under the hoods, but has a more user friendly experience.

Re: Brave Leo now uses Mixtral 8x7B as default

#155

Post by KeplerBoy »

dchichkov wrote: post #146
kkielhofner wrote: post #112

Earlier quoted context omitted.

I’ve worked with Jetson going back to the TK1 and I highly recommend you do not do this. Nvidia has significant dominance in the AI space because of their work on software and the overall platform. With the Jetson line being the sole exception. Use it for what it’s for - a targeted build for an embedded/specific application requiring small size and low power. The software is a mess. Support for Jetson (generally) is…

This is a bit surprising to hear. Current Jetpack 6 is Ubuntu 22.04 - this is the current Ubuntu LTS release. There's nothing ancient about it, no? I'm pretty sure, if I go and check versions of CUDA, PyTorch, Tensorflow - it'd be also relatively recent. I'd suggest checking what examples are available, see what community is doing, see if what you need had already been tried - https://www.jetson-ai-lab.com From what…

Yes, it's all rather recent in my experience. You get CUDA 12 and the newest Pytorch.

Re: Brave Leo now uses Mixtral 8x7B as default

#156

Post by jacooper »

trissi1996 wrote: post #138
jacooper wrote: post #133

Earlier quoted context omitted.

Supports* dedicated*

While not being officially supported, rocm runs just fine on my 6700XT, i just have to set an env var(export HSA_OVERRIDE_GFX_VERSION=10.3.0)

Really? Does everything run? Even AI stuff? Do you have any links where I can read more about that?

Re: Brave Leo now uses Mixtral 8x7B as default

#157

Post by gkbrk »

charcircuit wrote: post #153
danShumway wrote: post #149

Earlier quoted context omitted.

> My point was that "reduces ad blockers to static filter lists" is not true as it is not a static list of rules. That's not what we mean by static either. What we mean by static is that extensions lose the ability to dynamically analyze requests and block them using on-the-fly logic. > Yes, they are. Due to early versions of declarativeNetRequest only having a static list of rules there are a large amount of people…

>That's not what we mean by static either. What is a static filter list other than a list of filters that does not change? >When you replied to gkbrk saying that dynamic filters were supported you were not correcting misinformation I was saying that the filter list was dynamic and not static. >"Just inject scripts into the webpage" is a wild take I was pointing out the upper bounds of what was possible and not giving…

> What is a static filter list other than a list of filters that does not change?

A request is being made. At the time of the request, you go through a list of rules and decide if you're going to block or allow it.

Once the request reaches the filter, you are working with a static list because you cannot make decisions using the content or circumstances of that particular request. You cannot add or remove anything to your list after looking at the request.

A dynamic filter on the other hand could

- Resolve hostnames with DNS to do CNAME uncloaking

- Decode encoded requests so the website cannot send both real data and analytics/tracking data through a single URL. Being able to make dynamic decisions allows the ad blocker to inspect request headers and request bodies.

- Modify responses. The web page could be getting ads from a JSON endpoint with the rest of the page. You can use filterResponseData and change the response dynamically to strip the ads return an empty JSON list for example.

Re: Brave Leo now uses Mixtral 8x7B as default

#158

Post by Departed7405 »

danShumway wrote: post #140
davikr wrote: post #4

It's nice using Brave because you have Chromium's better performance, without having to worry about Manifest V2 dying and taking adblocking down with it. I have uBlock Origin enabled, but it has barely caught anything that slipped past the browser filters.

This has come up a few times, but as far as I know Gorhill is planning to full-on drop support for Chromium browsers outside of uBO Lite. Does Brave keeping Manifest V2 matter if developers stop maintaining the Chromium version of those extensions? I also still can't really find if Brave has an extension store or if it's connecting to Chrome's. If it's the latter, then it seems like V2 extensions are going away for B…

Brave indeed use the Chrome Web Store.

Re: Brave Leo now uses Mixtral 8x7B as default

#159

Post by mkesper »

stavros wrote: post #60
mkesper wrote: post #57

Earlier quoted context omitted.

Or a Linux machine with a Ryzen using the internal GPU and the unified RAM (scroll down at llama.cpp and look for ROCm).

What? I can do this? Runs to the PC EDIT: I cannot, I need to install ROCm to compile with it, and then install something called hipBLAS, and who knows what else.

Well, yes, you need to install ROCm and depdendencies. Have a look at https://rocm.docs.amd.com/projects/install-on-linux/en/lates... Debian trixie (not yet released) has most dependencies as packages. Or you can try a docker container https://rocm.docs.amd.com/projects/install-on-linux/en/lates...

Re: Brave Leo now uses Mixtral 8x7B as default

#160

Post by mkesper »

stavros wrote: post #74
assbuttbuttass wrote: post #72

Earlier quoted context omitted.

OpenCL should also work on AMD cards, and is way easier to install

Does that let me use unified memory on the GPU, though? Or is it just so I can use my CPU memory? EDIT: Oh, no, I have an nVidia GPU, AMD CPU.

I bet your AMD CPU has an internal GPU, too. That's what you can use with the unified memory.
Post reply on HN