Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Brave Leo now uses Mixtral 8x7B as default

brave.com

Post reply on HN
Print view View original

121–130 of 184 posts

Re: Brave Leo now uses Mixtral 8x7B as default

#121

Post by wolverine876 »

lolinder wrote: post #83
Dwedit wrote: post #75

Earlier quoted context omitted.

There is no way to confirm that claim, just like there is no way to confirm that a VPN service is "no log".

Yes, at some point if you're going to interface with other humans you will eventually just have to trust their word. For some people's threat models that isn't good enough, but for the vast majority of people—people who aren't being pursued by state intelligence agencies but who are squeamish about how much data a company like Google collects—a pinky promise from Brave or Mullvad is good enough.

> For some people's threat models that isn't good enough, but for the vast majority of people—people who aren't being pursued by state intelligence agencies but who are squeamish about how much data a company like Google collects—a pinky promise from Brave or Mullvad is good enough.

Who are you to say it's good enough (and ridicule people who disagree)? We don't have too much evidence of it, because they have very few options and of course most people are not informed and lack the expertise to understand the issues (a good situation for regulation). At one point lots of people used lead paint and were fine with it; they would have told us.

> Yes, at some point if you're going to interface with other humans you will eventually just have to trust their word.

There's technology, such as the authorization tokens used by Brave, that reduces that risk. Of course, no risk can be complete eliminated but that doesn't mean we shouldn't reduce it.

Re: Brave Leo now uses Mixtral 8x7B as default

#122

Post by KeplerBoy »

fsiefken wrote: post #111
KeplerBoy wrote: post #79

Earlier quoted context omitted.

Or a jetson orin agx (~2k$). Probably the cheapest way to get an Nvidia GPU with 64 GB of RAM.

I wonder what would be the cheapest way to run an LLM, with the latest Ryzen integrated graphics and 64G Ram or the Jetson AGX Orin 64. https://www.nvidia.com/en-us/autonomous-machines/embedded-sy...

The Ryzen is a lot cheaper, but most likely also a fair bit slower. You'd be looking at a 200$ CPU, 200$ Motherboard + 200$ of ddr5 ram. Throw in a case, nvme drive and power supply and you're still below $1k and those numbers are quite generous estimates, you could do it a lot cheaper by going AM4 with DDR4 ram.

Re: Brave Leo now uses Mixtral 8x7B as default

#123

Post by MuffinFlavored »

rhdunn wrote: post #2

If you want to run Mixtral 8x7B locally you can use llama.cpp (including with any of the supporting libraries/interfaces such as text-generation-webui) with https://huggingface.co/TheBloke/Nous-Hermes-2-Mixtral-8x7B-S... . The smallest quantized version (2bit) needs 20GB of RAM (which can be offloaded onto the VRAM of a decent 4090 GPU). The 4bit quantized versions are the largest models that can just about fit onto…

What differences would I measurably notice running the 2-bit version vs the 4-bit version vs the 6-bit vs the 8-bit?

Re: Brave Leo now uses Mixtral 8x7B as default

#124

Post by brucethemoose2 »

simonw wrote: post #114
brucethemoose2 wrote: post #93

Earlier quoted context omitted.

I've had less luck with Mixtral, but I run Yi 34B finetunes for general personal use, including quick queries for work. Its kinda like GPT 3.5, with no internet access and slightly less reliable responses, but unrestrained, much faster and with a huge (up to 75K on my Nvidia 3090) usable context. Mixtral is extremely fast though, at least at a batch size of 1.

Which Yi 34B finetunes are you using that have a 75,000 token length?

All of the Yi 200K finetunes should support it, but you have to be careful because some degrade the base model's quite excellent long context performance more than others. The very strong Bagel 34B DPO model, for instance, basically doesn't work at long context.

Nous Capybara is a popular one. I personally use my own merge of many models, and you can look through the constituent models to see if any interest you: https://huggingface.co/brucethemoose/Yi-34B-200K-DARE-megame...

You can't really use llama.cpp for super long context btw, its just too slow and vram inefficient at the moment.

Re: Brave Leo now uses Mixtral 8x7B as default

#125

Post by quinncom »

wolverine876 wrote: post #118

Kudos to Brave (for this and other privacy features): Unlinkable subscription: If you sign up for Leo Premium, you’re issued unlinkable tokens that validate your subscription when using Leo. This means that Brave can never connect your purchase details with your usage of the product, an extra step that ensures your activity is private to you and only you. The email you used to create your account is unlinkable to you…

This is very cool, and something I’d like to integrate in my own apps. Does anybody know how this works exactly, not using foreign keys?

Re: Brave Leo now uses Mixtral 8x7B as default

#126

Post by lolinder »

wolverine876 wrote: post #121
lolinder wrote: post #83

Earlier quoted context omitted.

Yes, at some point if you're going to interface with other humans you will eventually just have to trust their word. For some people's threat models that isn't good enough, but for the vast majority of people—people who aren't being pursued by state intelligence agencies but who are squeamish about how much data a company like Google collects—a pinky promise from Brave or Mullvad is good enough.

> For some people's threat models that isn't good enough, but for the vast majority of people—people who aren't being pursued by state intelligence agencies but who are squeamish about how much data a company like Google collects—a pinky promise from Brave or Mullvad is good enough. Who are you to say it's good enough (and ridicule people who disagree)? We don't have too much evidence of it, because they have very fe…

> say it's good enough (and ridicule people who disagree)?

I'm not ridiculing anyone, I explicitly say that for some people's threat models it isn't good enough.

Re: Brave Leo now uses Mixtral 8x7B as default

#127

Post by wolverine876 »

lolinder wrote: post #126
wolverine876 wrote: post #121

Earlier quoted context omitted.

> For some people's threat models that isn't good enough, but for the vast majority of people—people who aren't being pursued by state intelligence agencies but who are squeamish about how much data a company like Google collects—a pinky promise from Brave or Mullvad is good enough. Who are you to say it's good enough (and ridicule people who disagree)? We don't have too much evidence of it, because they have very fe…

> say it's good enough (and ridicule people who disagree)? I'm not ridiculing anyone, I explicitly say that for some people's threat models it isn't good enough.

You said that people who agreed with you are,

> people who aren't being pursued by state intelligence agencies

That implies that the only valid reason to disagree is if you are pursued by state intelligence agencies. Obviously it's ridiculous to think that you are.

Re: Brave Leo now uses Mixtral 8x7B as default

#128

Post by salad-tycoon »

wolverine876 wrote: post #127
lolinder wrote: post #126

Earlier quoted context omitted.

> say it's good enough (and ridicule people who disagree)? I'm not ridiculing anyone, I explicitly say that for some people's threat models it isn't good enough.

You said that people who agreed with you are, > people who aren't being pursued by state intelligence agencies That implies that the only valid reason to disagree is if you are pursued by state intelligence agencies. Obviously it's ridiculous to think that you are.

Well, these days a lot of people are actually pursued. You know like parents, social activists, various religious movements. It’s not just only the Dr. Evils of the world anymore.

Re: Brave Leo now uses Mixtral 8x7B as default

#129

Post by charcircuit »

HeatrayEnjoyer wrote: post #102
rpastuszak wrote: post #67

Earlier quoted context omitted.

It makes them almost useless in practice.

Because the filter list is capped, right? Is there a reason the Brave team cannot just remove or increase the cap?

>Because the filter list is capped, right?

The limits are 300,000 static rules [1] + 30,000 dynamic rules [2] + 5,000 session rules [3]. For reference easylist is about 35k rules. The Chrome team has been constantly tweaking these limits themselves and Brave could set their own limits if they wish. The API is designed such that extensions can query to see how many rules they can use.

[1] https://source.chromium.org/chromium/chromium/src/+/main:ext...

[2] https://source.chromium.org/chromium/chromium/src/+/main:out...

[3] https://source.chromium.org/chromium/chromium/src/+/main:out...

Re: Brave Leo now uses Mixtral 8x7B as default

#130

Post by charcircuit »

gkbrk wrote: post #110
HeatrayEnjoyer wrote: post #102

Earlier quoted context omitted.

Because the filter list is capped, right? Is there a reason the Brave team cannot just remove or increase the cap?

Not just because of the filter list cap. It also reduces ad blockers to static filter lists instead of powerful dynamic filters. MV3 makes it impossible for ad-blockers to inspect requests with code and then allow/deny dynamically.

>It also reduces ad blockers to static filter lists instead of powerful dynamic filters.

This is very outdated information and borderline misinformation by representing it as how it currently works. It allows for 30,000 dynamic rules and 5,000 session rules (session rules only persist until the browser is closed).

>MV3 makes it impossible for ad-blockers to inspect requests with code and then allow/deny dynamically.

Giving this ability to extensions can slow down the browser for the user. These ads can still be blocked through other means.

Post reply on HN