Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The Prompt API

developer.chrome.com

Post reply on HN
Print view View original

81–90 of 157 posts

Re: The Prompt API

#81

Post by meander_water »

domenicd wrote: post #77
meander_water wrote: post #74

This looks like it uses Gemini Nano under the hood. But the latest Gemma4 E2B and E4B models appear to be much better, so you'd probably be better off deploying quantized versions through an extension for now. - Gemini Nano-1: 46% MMLU, 1.8B - Gemini Nano-2: 56% MMLU, 3.25B - Gemma4 E2B: 60.0% MMLU, 2.3B - Gemma4 E4B: 69.4% MMLU, 4.5B Sources: - https://huggingface.co/google/gemma-4-E2B-it - https://android-developer…

I no longer have any inside knowledge, but from my time on this team they were very quick about getting the latest small (Google) models into Chrome. I expect that if Gemma 4 (or its equivalent Gemini Nano) isn't already in Chrome, then it will be soon. Note that the article here was last updated 2025-09-21, and as of that time it was already on Gemini Nano 3.

Thanks for the insider info! Do you know if there are any published benchmarks for Nano 3?

Re: The Prompt API

#82

Post by zozbot234 »

avaer wrote: post #4

It works, I've shipped this as a "local inference"/poor person's ollama for low-end llm tasks like search. The main win is that it's free and privacy preserving, and (mostly) transparent to users in that they don't have to do anything, which is great for giving non-technical users local inference without making them do scary native things. But keep in mind the actual experience for users is not great; the model downl…

> But keep in mind the actual experience for users is not great; the model download is orders of magnitude greater than downloading the browser itself, and something that needs to happen before you get your first token back.

With MoE models, you could fetch expert layers from the network on demand by issuing HTTP range queries for the corresponding offset, similar to how bittorrent downloads file chunks from multiple hosts. You'd still have to download shared layers, but time to first token would now be proportional to active-size rather than total-size. Of course this wouldn't be totally "offline" inference anymore, but for a web browser feature that's not a key consideration.

Re: The Prompt API

#83

Post by NitpickLawyer »

zozbot234 wrote: post #82
avaer wrote: post #4

It works, I've shipped this as a "local inference"/poor person's ollama for low-end llm tasks like search. The main win is that it's free and privacy preserving, and (mostly) transparent to users in that they don't have to do anything, which is great for giving non-technical users local inference without making them do scary native things. But keep in mind the actual experience for users is not great; the model downl…

> But keep in mind the actual experience for users is not great; the model download is orders of magnitude greater than downloading the browser itself, and something that needs to happen before you get your first token back. With MoE models, you could fetch expert layers from the network on demand by issuing HTTP range queries for the corresponding offset, similar to how bittorrent downloads file chunks from multiple…

> With MoE models, you could fetch expert layers from the network on demand

This is a common misconception, probably due to the unfortunate naming. Expert layers are not "expert" at any particular subject, and active-size only refers to the activated layers per token. You'd still need all (or most of all) the layers for any particular query, even if some layers have a very low chance of being activated.

All in all, you'd be better off with lazy loading the entire model, at least you'd know you have the capability to run inference from then on.

Re: The Prompt API

#84

Post by niek_pas »

encrux wrote: post #57
haberman wrote: post #16

This API seems perfect for an idea I've had for a while: a de-snarkifier for social media. Social media can be intellectually stimulating and educational, but it's also easy to get sucked into ideological sniping and flamewars, even if you didn't go looking for it. The emotional and intellectual energy spent flaming strangers on the Internet is a complete waste of human capital. With an API like this, I assume you co…

For YouTube, this already exists and I‘m using it. The extension is caller DeArrow and aims to reduce sensationalism via crowdsourcing, though I wouldn’t be surprised if top contributors are bots using LLMs.

Man, that before-after slider on the home page makes me so sad... YouTube used to just be random people sharing cool stuff, and those de-sensationalized titles really brought me back to that time for a second! Cool stuff.

Re: The Prompt API

#85

Post by redsocksfan45 »

haberman wrote: post #16

This API seems perfect for an idea I've had for a while: a de-snarkifier for social media. Social media can be intellectually stimulating and educational, but it's also easy to get sucked into ideological sniping and flamewars, even if you didn't go looking for it. The emotional and intellectual energy spent flaming strangers on the Internet is a complete waste of human capital. With an API like this, I assume you co…

[dead]

Re: The Prompt API

#86

Post by fg137 »

jfoster wrote: post #63
cdrini wrote: post #20

Earlier quoted context omitted.

True, but arguably better than "sorry, to use our website, you must have a ChatGPT subscription."

Also much better than every website wanting its own 22 GB rather than the 22 GB being a shared resource.

I would very much like not to have to download 22 GB for some inference capability that is way worse than API calls both in terms of quality and speed.

I would rather pay money than seeing this thing running in my browser that only prints 5 tps on high-end consumer hardware.

Re: The Prompt API

#87

Post by fg137 »

cdrini wrote: post #20
fg137 wrote: post #2

"sorry, to use our website, you must have at least 22 GB of free disk space."

True, but arguably better than "sorry, to use our website, you must have a ChatGPT subscription."

More like "you need to sign up for our website and pay for a subscription", and I'd much rather do that if it's actually providing value. I am absolutely not going to run model locally which slowly churns out words at 5 tps while making the computer hot to touch.

Re: The Prompt API

#88

Post by zozbot234 »

NitpickLawyer wrote: post #83
zozbot234 wrote: post #82

Earlier quoted context omitted.

> But keep in mind the actual experience for users is not great; the model download is orders of magnitude greater than downloading the browser itself, and something that needs to happen before you get your first token back. With MoE models, you could fetch expert layers from the network on demand by issuing HTTP range queries for the corresponding offset, similar to how bittorrent downloads file chunks from multiple…

> With MoE models, you could fetch expert layers from the network on demand This is a common misconception, probably due to the unfortunate naming. Expert layers are not "expert" at any particular subject, and active-size only refers to the activated layers per token. You'd still need all (or most of all) the layers for any particular query, even if some layers have a very low chance of being activated. All in all, y…

Ultimately it would amount to lazy-loading the model, but the parameters themselves would be fetched from the network as needed, which still decreases time-to-first-token. It's true that "expert" choices will span most of the model, regardless of any particular "subject" or "topic" choice, but if we simply care about time-to-first-token it's still a viable strategy.

Re: The Prompt API

#89

Post by sebzim4500 »

encrux wrote: post #57
haberman wrote: post #16

This API seems perfect for an idea I've had for a while: a de-snarkifier for social media. Social media can be intellectually stimulating and educational, but it's also easy to get sucked into ideological sniping and flamewars, even if you didn't go looking for it. The emotional and intellectual energy spent flaming strangers on the Internet is a complete waste of human capital. With an API like this, I assume you co…

For YouTube, this already exists and I‘m using it. The extension is caller DeArrow and aims to reduce sensationalism via crowdsourcing, though I wouldn’t be surprised if top contributors are bots using LLMs.

For people like me had tried it in the past and found it annoying, note that it now has a 'casual' mode where it only changes the truly useless titles and leaves reasonable ones alone.

Re: The Prompt API

#90

Post by tom1337 »

The idea of having local LLMs accessible in the browser for privacy concerning is nice i guess but when each browser has a different model attached to this API testing becomes even more a nightmare then now. I wonder if this will drive more users towards chrome because most of the usages of this API might be just tailored to fit the Gemini Nano model?
Post reply on HN