The Prompt API

"sorry, to use our website, you must have at least 22 GB of free disk space."

It works, I've shipped this as a "local inference"/poor person's ollama for low-end llm tasks like search. The main win is that it's free and privacy preserving, and (mostly) transparent to users in that they don't have to do anything, which is great for giving non-technical users local inference without making them do scary native things.

But keep in mind the actual experience for users is not great; the model download is orders of magnitude greater than downloading the browser itself, and something that needs to happen before you get your first token back. That's unfixable until operating systems start reliably shipping their own prebaked models that an API like this could plug into.

[dead]

Seems like a good way for a rogue JS script to offload token generation to a bunch of unsuspecting visitors

It would actually be pretty interesting to see if its possible to decentralize the compute to generate something useful from a larger prompt broken down and sent to a bunch of browsers using a subagent pattern or something like RLM, each working on a smaller part of the prompt

> That's unfixable until operating systems start reliably shipping their own prebaked models that an API like this could plug into.

Maybe the next big thing will be some software subscription premium offers with a bunch of 5090s as an extra.