Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The Prompt API

developer.chrome.com

Post reply on HN
Print view View original

111–120 of 157 posts

Re: The Prompt API

#111

Post by gruez »

2ndorderthought wrote: post #108
gruez wrote: post #106

Earlier quoted context omitted.

>I can see applications for this being things like "while device is at rest and charging summarize all of the users recent text communications" or whatever else as a legal loop hole for wiretap laws This just exposes an API for sites to use. If they wanted to do the types of spying you're cynically suggesting, they could just add it without an API and you'd be none the wiser. Chrome contains closed source components…

It's a lot easier to hide the language they need in a EULA for a feature like this than it would be elsewhere. I appreciate you feel this is a cynical take. But have you seen the class action lawsuits against Google over the last 5 years? They exceed a billion dollars as far as I can remember and they are for more blatant things than this.

>It's a lot easier to hide the language they need in a EULA for a feature like this than it would be elsewhere.

Why would adding a ML API or library require an EULA change?

Re: The Prompt API

#112

Post by bfeist »

haberman wrote: post #16

This API seems perfect for an idea I've had for a while: a de-snarkifier for social media. Social media can be intellectually stimulating and educational, but it's also easy to get sucked into ideological sniping and flamewars, even if you didn't go looking for it. The emotional and intellectual energy spent flaming strangers on the Internet is a complete waste of human capital. With an API like this, I assume you co…

I would love an app like this. I am a frequent user of https://www.boringreport.org/ for news, which does something like what you’re describing but for news articles.

Re: The Prompt API

#113

Post by ceejayoz »

meander_water wrote: post #74

This looks like it uses Gemini Nano under the hood. But the latest Gemma4 E2B and E4B models appear to be much better, so you'd probably be better off deploying quantized versions through an extension for now. - Gemini Nano-1: 46% MMLU, 1.8B - Gemini Nano-2: 56% MMLU, 3.25B - Gemma4 E2B: 60.0% MMLU, 2.3B - Gemma4 E4B: 69.4% MMLU, 4.5B Sources: - https://huggingface.co/google/gemma-4-E2B-it - https://android-developer…

> This looks like it uses Gemini Nano under the hood.

Yes; "With the Prompt API, you can send natural language requests to Gemini Nano in the browser."

Re: The Prompt API

#114

Post by contagiousflow »

whatarethembits wrote: post #95
jurgenburgen wrote: post #30

Earlier quoted context omitted.

On the other hand it would make all comments sound the same and further dilute internet content into average slop.

I'm hoping that something like this can condense a 1000+ comments thread to couple of paragraphs at most.

Why would you want that?

Re: The Prompt API

#115

Post by jfoster »

fg137 wrote: post #86
jfoster wrote: post #63

Earlier quoted context omitted.

Also much better than every website wanting its own 22 GB rather than the 22 GB being a shared resource.

I would very much like not to have to download 22 GB for some inference capability that is way worse than API calls both in terms of quality and speed. I would rather pay money than seeing this thing running in my browser that only prints 5 tps on high-end consumer hardware.

Why are you pretending those are the options?

The options are:

1. 22GB per website

2. 22GB per browser

3. 0GB / No AI capabilities

By having this in Chrome they are simply ensuring that option 2 replaces option 1. You still have option 3.

Re: The Prompt API

#116

Post by masfuerte »

gruez wrote: post #106
2ndorderthought wrote: post #97

Earlier quoted context omitted.

Is it actually privacy preserving? Chrome mostly exists to extract all the information from a user it can without immediately getting a lawsuit of greater penalty than what is gained through ads, military contracts, etc. Android isn't too far off either. I would welcome any alternative to this. I can see applications for this being things like "while device is at rest and charging summarize all of the users recent te…

>I can see applications for this being things like "while device is at rest and charging summarize all of the users recent text communications" or whatever else as a legal loop hole for wiretap laws This just exposes an API for sites to use. If they wanted to do the types of spying you're cynically suggesting, they could just add it without an API and you'd be none the wiser. Chrome contains closed source components…

Do you think no-one would notice that the Chrome download was 20GB larger?

Re: The Prompt API

#117

Post by bavell »

zozbot234 wrote: post #88
NitpickLawyer wrote: post #83

Earlier quoted context omitted.

> With MoE models, you could fetch expert layers from the network on demand This is a common misconception, probably due to the unfortunate naming. Expert layers are not "expert" at any particular subject, and active-size only refers to the activated layers per token. You'd still need all (or most of all) the layers for any particular query, even if some layers have a very low chance of being activated. All in all, y…

Ultimately it would amount to lazy-loading the model, but the parameters themselves would be fetched from the network as needed, which still decreases time-to-first-token. It's true that "expert" choices will span most of the model, regardless of any particular "subject" or "topic" choice, but if we simply care about time-to-first-token it's still a viable strategy.

Perhaps you could generate a few tokens before the entire model is downloaded, but since every token takes a potentially different "path" through an MoE model, you'd still need to wait for the entire download before getting deeper than a handful of tokens... which is not really a UX improvement imo.

Re: The Prompt API

#118

Post by Tepix »

meander_water wrote: post #74

This looks like it uses Gemini Nano under the hood. But the latest Gemma4 E2B and E4B models appear to be much better, so you'd probably be better off deploying quantized versions through an extension for now. - Gemini Nano-1: 46% MMLU, 1.8B - Gemini Nano-2: 56% MMLU, 3.25B - Gemma4 E2B: 60.0% MMLU, 2.3B - Gemma4 E4B: 69.4% MMLU, 4.5B Sources: - https://huggingface.co/google/gemma-4-E2B-it - https://android-developer…

The Prompt API uses the model that's available in your browser. For Edge I believe it's Phi4.

Re: The Prompt API

#119

Post by Tepix »

avaer wrote: post #4

It works, I've shipped this as a "local inference"/poor person's ollama for low-end llm tasks like search. The main win is that it's free and privacy preserving, and (mostly) transparent to users in that they don't have to do anything, which is great for giving non-technical users local inference without making them do scary native things. But keep in mind the actual experience for users is not great; the model downl…

It's a one-time download shared by all websites that use the Prompt API.

What's a bigger issue is that the models on most standard PCs are both tiny and slow. I was going to try using the Prompt API to change the text of (infocom) text adventures on the fly. But for many PCs, this will currently be too slow to be feasible.

Re: The Prompt API

#120

Post by Tepix »

jameslk wrote: post #8

Seems like a good way for a rogue JS script to offload token generation to a bunch of unsuspecting visitors It would actually be pretty interesting to see if its possible to decentralize the compute to generate something useful from a larger prompt broken down and sent to a bunch of browsers using a subagent pattern or something like RLM, each working on a smaller part of the prompt

token generation of a tiny model. Hardly worth anything.
Post reply on HN