Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The Prompt API

developer.chrome.com

Post reply on HN
Print view View original

61–70 of 157 posts

Re: The Prompt API

#61

Post by jfoster »

subhobroto wrote: post #11
avaer wrote: post #4

It works, I've shipped this as a "local inference"/poor person's ollama for low-end llm tasks like search. The main win is that it's free and privacy preserving, and (mostly) transparent to users in that they don't have to do anything, which is great for giving non-technical users local inference without making them do scary native things. But keep in mind the actual experience for users is not great; the model downl…

> It works, I've shipped this as a "local inference"/poor person's ollama for low-end llm tasks like search fantastic! > the model download is orders of magnitude greater than downloading the browser itself, and something that needs to happen before you get your first token back sure but does this mean the model is lazily downloaded? that is, if I used this and I am the first time the model was called, the user would…

Doesn't sound great, but consider how much better this is than every webpage trying to load their own models.

If it turns out useful enough I'm sure browsers will just start including it as (perhaps optional?) part of installation.

Re: The Prompt API

#63

Post by jfoster »

cdrini wrote: post #20
fg137 wrote: post #2

"sorry, to use our website, you must have at least 22 GB of free disk space."

True, but arguably better than "sorry, to use our website, you must have a ChatGPT subscription."

Also much better than every website wanting its own 22 GB rather than the 22 GB being a shared resource.

Re: The Prompt API

#65

Post by flashdesk »

netcan wrote: post #54
jychang wrote: post #50

Earlier quoted context omitted.

Yeah, I 100% agree with the caution in this comment. I see the merit in such a proposal. It's the linguistic equivalent to boiling the food you consume, instead of eating it raw with all the associated bad stuff. The problem is, as you said, that this plan is unlikely to be as rosy as it's portrayed and probably has a lot of drawbacks in real life. Interesting to think about and explore, though.

I wasn't even talking about drawbacks, though that applies too. I mean... you would be basically taking a complex thing, transforming and reconstructing it. What we want out of social media isn't a simple, legible function. The positives. You'd have to discover them. If someone starts building with the intitial idea above, my guess is that they'd end up with some sort of custom feed that draws inspiration and inputs…

[dead]

Re: The Prompt API

#67

Post by paganel »

avaer wrote: post #4

It works, I've shipped this as a "local inference"/poor person's ollama for low-end llm tasks like search. The main win is that it's free and privacy preserving, and (mostly) transparent to users in that they don't have to do anything, which is great for giving non-technical users local inference without making them do scary native things. But keep in mind the actual experience for users is not great; the model downl…

> operating systems start reliably shipping their own prebaked models

Here's to hoping that that dystopia will never happen.

Re: The Prompt API

#68

Post by danpalmer »

subhobroto wrote: post #59
tastroder wrote: post #36

Earlier quoted context omitted.

chrome://on-device-internals reports "Model Name: v3Nano Version: 2025.06.30.1229 Folder size: 4,072.13 MiB" on a random Windows machine I just checked.

Thank You stranger! I would have assumed the size would vary based on whether your hardware supports the high-quality GPU backend (4 GB) or defaults to a smaller CPU-compatible version (3 GB) but the 22GB note on that page is really confusing. Even if it was including the model server where's the remaining 18GB going towards?

I'd imagine that the 22GB was decided through modelling various scenarios. For a start, it's not just a 4GB current model, it's 2x4GB to be able to update it without needing time when the computer is without a model, that's up to 8GB.

Then it's possible the model you get will scale with the CPU/GPU/RAM available, so if you have a 12GB GPU you probably get a better model, perhaps that's a 10-11GB model? At 2x that's 22GB.

Then consider that a machine is not static, GPUs/hardware come and go, VRAM allocation in integrated graphics changes, etc. You end up with just needing to pick a number and not confuse users.

Re: The Prompt API

#69

Post by comboy »

domenicd wrote: post #62

I led the design effort on this API, before retiring. Here's my writeup on some of the considerations that went into it: https://domenic.me/builtin-ai-api-design/

How do you envision short term and long term target usage of it?

And do you guys communicate between other browsers when doing something like this to try to settle on something common? I don't mean W3C but practically, it's a small world after all.

Post reply on HN