Chrome is adding `window.ai` – a Gemini Nano AI model right inside the browser • Hacker News Board

Re: Chrome is adding `window.ai` – a Gemini Nano AI model right inside the browser

#111

Post by niutech » Sun, Jun 30, 2024, 7:02 AM UTC

SquareWheel wrote: post #108

So it's loading an instruct model for inference? That seems a fair bit less useful than a base model, at least for more advanced use cases. What about running LoRAs, adjusting temperature, configuring prompt templates, etc? It seems pretty early to build something like this into the browser. The technology is still changing so rapidly, it might look completely different in 5 years. I'm a huge fan of local AI, and of…

The base model can be found on HF (https://huggingface.co/wave-on-discord/gemini-nano) and run in any web browser using MediaPipe on WebGPU: https://x.com/niu_tech/status/1807073666888266157

As for temperature and topK, you can set them in the AITextSessionOptions object as an argument to `window.ai.createTextSession(options)` (source: https://source.chromium.org/chromium/chromium/src/+/main:thi...)

You should also be able to set it by adding the switches: `chrome --args --enable-features=OptimizationGuideOnDeviceModel:on_device_model_temperature/0.5/on_device_model_topk/8` (source: https://issues.chromium.org/issues/339471377#comment12)

The default temperature is 0.8 and default topK is 3 (source: https://source.chromium.org/chromium/chromium/src/+/main:com...)

As for LoRA, Google will provide a Fine-Tuning (LoRA) API in Chrome: https://developer.chrome.com/docs/ai/built-in#browser_archit...

Re: Chrome is adding `window.ai` – a Gemini Nano AI model right inside the browser

#112

Post by kinlan » Sun, Jun 30, 2024, 7:06 AM UTC

flakiness wrote: post #73

So they don't standardize things anymore? Look at WebNN [1]. It's from Microsoft and is basically DirecttML but they at least pretend to make it a Web thing. The posture matters. Apple tried to expose Metal through WebGPU [2] then silent-abandoned it. But they had the posture, and other vendors picked it up and made it real. That won't happen to window.ai until they stop sleepwalking. [1] https://www.w3.org/TR/webnn/…

It's a very experimental API and that's why it's only behind a flag and not available to the general web for people to use. We will be taking these through the standards process (e.g the higher level translate API - https://github.com/WICG/translation-api)

Re: Chrome is adding `window.ai` – a Gemini Nano AI model right inside the browser

#113

Post by langsoul-com » Sun, Jun 30, 2024, 7:07 AM UTC

Man, the bloat is unreal. Can't we just have a browser without all the extra crap?

Re: Chrome is adding `window.ai` – a Gemini Nano AI model right inside the browser

#114

Post by tasuki » Sun, Jun 30, 2024, 7:26 AM UTC

I wish AI came for my Android keyboard.

I regularly type in English, Czech, and Polish, and Gboard doesn't even know some of the basic words or word forms.

Re: Chrome is adding `window.ai` – a Gemini Nano AI model right inside the browser

#115

Post by thiht » Sun, Jun 30, 2024, 7:30 AM UTC

haolez wrote: post #101

On a side note, I think window.assistant might be a better name. AI is a tired and ambiguous term at this point.

Yes, window.ai is a terrible name

Re: Chrome is adding `window.ai` – a Gemini Nano AI model right inside the browser

#116

Post by thiht » Sun, Jun 30, 2024, 7:33 AM UTC

darepublic wrote: post #92

lolinder wrote: post #65

> The code below is all you need to stream text with Chrome AI and the Vercel AI SDK. ... `chromeai` implements a Provider that uses `window.ai` under the hood Leave it to Vercel to announce `window.ai` on Google's behalf by showing off their own abstraction but not the actual Chrome API. Here's a blog post from a few days ago that shows how the actual `window.ai` API works [0]. The code is extremely simple and reall…

web dev is rife with this stuff. wrappers upon wrappers with a poor trade off between adding api overhead / obscuring the real workings of what's going on and any actual enhanced functionality or convenience. it's done for github stars and rep.

Or maybe it’s done because people like to try and experiment new things, see what works and what doesn’t, with sometimes surprising results. I thought the name of this site was Hacker News, let people do weird things

Re: Chrome is adding `window.ai` – a Gemini Nano AI model right inside the browser

#117

Post by j10u » Sun, Jun 30, 2024, 7:35 AM UTC

simonw wrote: post #107

If this is the API that Google are going with here: const model = await window.ai.createTextSession(); const result = await model.prompt("3 names for a pet pelican"); There's a VERY obvious flaw: is there really no way to specify the model to use? Are we expecting that Gemini Nano will be the one true model, forever supported by this API baked into the world's most popular browser? Given the rate at which models are…

I'm pretty sure that with time, they will be forced to let users choose the model. Just like it happened with the search engine...

Re: Chrome is adding `window.ai` – a Gemini Nano AI model right inside the browser

#118

Post by Kwpolska » Sun, Jun 30, 2024, 7:43 AM UTC

simonw wrote: post #107

If this is the API that Google are going with here: const model = await window.ai.createTextSession(); const result = await model.prompt("3 names for a pet pelican"); There's a VERY obvious flaw: is there really no way to specify the model to use? Are we expecting that Gemini Nano will be the one true model, forever supported by this API baked into the world's most popular browser? Given the rate at which models are…

Since when can you expect stability with random bullshit generators? They are constantly changed, and they involve a lot of randomness.

Re: Chrome is adding `window.ai` – a Gemini Nano AI model right inside the browser

#119

Post by luke-stanley » Sun, Jun 30, 2024, 7:47 AM UTC

simonw wrote: post #107

If this is the API that Google are going with here: const model = await window.ai.createTextSession(); const result = await model.prompt("3 names for a pet pelican"); There's a VERY obvious flaw: is there really no way to specify the model to use? Are we expecting that Gemini Nano will be the one true model, forever supported by this API baked into the world's most popular browser? Given the rate at which models are…

Presumably something like model.includes("gemini-nano:0.4") could work?

Re: Chrome is adding `window.ai` – a Gemini Nano AI model right inside the browser

#120

Post by makeitdouble » Sun, Jun 30, 2024, 7:47 AM UTC

visarga wrote: post #103

makeitdouble wrote: post #54

Earlier quoted context omitted.

I share a bit of parent's skepticism: the possibilities are infinite （as many things really), but do we need to dive in head first and sprinkle "AI" dust everywhere just in case some of it could be useful ? For instance I don't need my browser to pass the Turing test. I might need better filtering and better search, but it also doesn't need to be baked in the browser. Your analogy to electricity is interesting: do yo…

Filtering, search, summarization, reranking, and security protection (pishing, data leaks) - the necessary functionality adds up

These are all currently independent plugins or applications that operate partly regardless of the browser they target.

In particular I get to choose the best options for each of them (in particular search, filtering and security being independent from each other seems like a core requirement to me). The most telling part to me is how extensions come and go, and we move on from one to the other. The same kind of rollover won't be an option with everything in Apple's AI for instance.

This could come down the divide between the Unix philosophy of a constellation of specialized tools working together or a huge monolith responsible for everything.

I don't see the latter as a viable approach at scale.