Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Mistral X Mozilla: Private, Multilingual AI Browsing

mistral.ai

Post reply on HN
Print view View original

101–110 of 211 posts

Re: Mistral X Mozilla: Private, Multilingual AI Browsing

#101

Post by peri-cl »

walrus01 wrote: post #47
peri-cl wrote: post #5

This is an excellent use case for completely local, small model inference, yet for inexplicable reasons Mozilla wants to normalize uploading your entire private browsing history to a cloud. These (Mistral's and Mozilla's) marketing pages aren't candid enough to clearly explain the difference between local and cloud inference, and that they're asking you to consent to enabling the latter. I'd call that the bare minimu…

I am not trying to defend Mozilla doing this, and I don't support sending data to cloud based services like this in a way that users won't understand. But I also think that the state of the art in small LLM and user device capabilities aren't there yet to put a "good enough to be actually useful" local-only LLM as a prepackaged thing in a mass market distributed browser. You don't want a browser that takes 10GB of ex…

Well, it depends on the task, doesn't it? "running shoes I looked at last week" / "Here's what I found in your browsing history:" doesn't need a 119 billion parameter frontier model; it's a RAG problem for the 0.6 B embedding models. That's an example Mozilla offers. Presumably to explain to their users why it's essential they hand over their last week's browsing history for this convenience (but it isn't! Hardly for that!)

I feel it's wrong to tell users that it's important and normal to relinquish all control of their—extremely personal—life history, in bulk, in plaintext, to strangers.

I agree wholeheartedly that remote server inference is super useful, and that local inference falls far short on many tasks. (I have no objection at all to Mozilla providing a cloud inference feature).

What I don't buy is that we must ask users to redraw their personal boundaries so that their most intimate life details, and remote frontier-model inference, overlap. They do not need to overlap.

You can accomplish a lot with private local inference with the smallest of models; and you can accomplish a lot on remote servers which aren't privy to everything. If some convenience is lost by not combining the two, well, so be it. I'm sure most people would agree, if all of this was laid out plainly.

Re: Mistral X Mozilla: Private, Multilingual AI Browsing

#102

Post by mossTechnician »

nhinck3 wrote: post #73
peri-cl wrote: post #5

This is an excellent use case for completely local, small model inference, yet for inexplicable reasons Mozilla wants to normalize uploading your entire private browsing history to a cloud. These (Mistral's and Mozilla's) marketing pages aren't candid enough to clearly explain the difference between local and cloud inference, and that they're asking you to consent to enabling the latter. I'd call that the bare minimu…

Anyone else's extensions not work on support.mozilla.org?

You can disable this in about:config by modifying extensions.webextensions.restrictedDomains. It's probably there to help prevent malicious extensions from hijacking official browser support. I reckon many people here don't need it.

Re: Mistral X Mozilla: Private, Multilingual AI Browsing

#105

Post by [deleted] »

peri-cl wrote: post #5

This is an excellent use case for completely local, small model inference, yet for inexplicable reasons Mozilla wants to normalize uploading your entire private browsing history to a cloud. These (Mistral's and Mozilla's) marketing pages aren't candid enough to clearly explain the difference between local and cloud inference, and that they're asking you to consent to enabling the latter. I'd call that the bare minimu…

[deleted]

Re: Mistral X Mozilla: Private, Multilingual AI Browsing

#106

Post by walrus01 »

peri-cl wrote: post #101
walrus01 wrote: post #47

Earlier quoted context omitted.

I am not trying to defend Mozilla doing this, and I don't support sending data to cloud based services like this in a way that users won't understand. But I also think that the state of the art in small LLM and user device capabilities aren't there yet to put a "good enough to be actually useful" local-only LLM as a prepackaged thing in a mass market distributed browser. You don't want a browser that takes 10GB of ex…

Well, it depends on the task, doesn't it? "running shoes I looked at last week" / "Here's what I found in your browsing history:" doesn't need a 119 billion parameter frontier model; it's a RAG problem for the 0.6 B embedding models. That's an example Mozilla offers. Presumably to explain to their users why it's essential they hand over their last week's browsing history for this convenience (but it isn't! Hardly for…

I'm of the opinion that local inference should be done to the greatest extent that is realistically possible, at the earlier time that the hardware/average user platform is capable of doing so. I personally spend a fair bit on kWh extra in my home electrical bill monthly for having a good sized chunk of local inference ability in my house, but that's not a common thing yet.

If mozilla is doing things to send users down the path of doing this externally, they need to be much more upfront and transparent with the users about where their data is going, and not bury it in some terms/conditions that only nerds will hunt for.

Re: Mistral X Mozilla: Private, Multilingual AI Browsing

#108

Post by vgalin »

dwedge wrote: post #104

These are two companies that seem to be slipping in ethics recently. I'd prefer this over a chrome and openai offering, but I no longer trust these two to necessarily do the right thing regarding privacy

> These are two companies that seem to be slipping in ethics recently

What makes you say that?

Re: Mistral X Mozilla: Private, Multilingual AI Browsing

#109

Post by nullc »

walrus01 wrote: post #47
peri-cl wrote: post #5

This is an excellent use case for completely local, small model inference, yet for inexplicable reasons Mozilla wants to normalize uploading your entire private browsing history to a cloud. These (Mistral's and Mozilla's) marketing pages aren't candid enough to clearly explain the difference between local and cloud inference, and that they're asking you to consent to enabling the latter. I'd call that the bare minimu…

I am not trying to defend Mozilla doing this, and I don't support sending data to cloud based services like this in a way that users won't understand. But I also think that the state of the art in small LLM and user device capabilities aren't there yet to put a "good enough to be actually useful" local-only LLM as a prepackaged thing in a mass market distributed browser. You don't want a browser that takes 10GB of ex…

Have you tried Ling-3.0-tiny? It runs fine on CPU-- on a 14700KF gets 40tg/s and 250pp/s and on a ordinary gpu (RTX 4070) does over 200tg/s with no MTP and 7185pp/s. (my figures are Q8, though presumably a good Q4 would be faster)

It's certainly not as capable as something that needs a high memory gpu for quick performance, but I was quite impressed with it for what it is.

(and fwiw, I had it translate your last paragraph to German, then used google translate back to english: "I wish they had handled this clearly and transparently via an opt-in mechanism—not enabled by default—that explains what Mistral is (not a major American cloud company, but a relatively small French startup) and that your prompts and LLM activities are sent to their servers. I also wish there were documentation explaining how the data is handled and stored in a way that inspires trust.").

Re: Mistral X Mozilla: Private, Multilingual AI Browsing

#110

Post by walrus01 »

nullc wrote: post #109
walrus01 wrote: post #47

Earlier quoted context omitted.

I am not trying to defend Mozilla doing this, and I don't support sending data to cloud based services like this in a way that users won't understand. But I also think that the state of the art in small LLM and user device capabilities aren't there yet to put a "good enough to be actually useful" local-only LLM as a prepackaged thing in a mass market distributed browser. You don't want a browser that takes 10GB of ex…

Have you tried Ling-3.0-tiny? It runs fine on CPU-- on a 14700KF gets 40tg/s and 250pp/s and on a ordinary gpu (RTX 4070) does over 200tg/s with no MTP and 7185pp/s. (my figures are Q8, though presumably a good Q4 would be faster) It's certainly not as capable as something that needs a high memory gpu for quick performance, but I was quite impressed with it for what it is. (and fwiw, I had it translate your last para…

How much RAM does it take up in total? I'll have to give that a try on one of my test systems. Looking at a somewhat randomly chose GGUF quantization of it, looks like just under 5GB on disk in Q4, so RAM usage somewhere around 5-6GB?

https://huggingface.co/bartowski/Ling-3.0-tiny-GGUF

Post reply on HN