WebLLM: high-performance in-browser LLM inference engine

Project is de facto dead, used it for many years and had to rip it out 6 months ago, don't waste your time.

We use the ONNX runtime for small models in the browser https://github.com/microsoft/onnxruntime

This seems to be the demo:

https://chat.webllm.ai/

I am getting:

    WebGPUNotAvailableError: WebGPU is not supported in
    your current environment, but it is necessary to
    run the WebLLM engine.

On both, FireFox and Chromium on Linux.

You can enable WebGPU support in Google Chrome by turning on hardware acceleration and activating the WebGPU flag. It Works.

There's, quietly, a llama.cpp WebGPU backend that works *great*.

Some hacking required, it's unsupported, a side project for one of the lead maintainers and someone in school.

Note of caution, llama.cpp isn't what it was, the grunt-level maintainers are left to their own devices. There's one key subsystem where things break regularly and the engineering is poor, and the "lead maintainer" is aggro and isn't really involved after delivering their big refactor that was DOA, other than telling people on issues he's never seen their repro himself. Carefully pick models you can support down to "I can patch around the Jinja template engine". Go through the same llama web server APIs. c.f. github / telosnex / fllama if you need a reference

(n.b. seeing peer comments, its much better than ONNX, ONNX never got within spitting distance of llama.cpp, my understanding from watching the LLM runtime is its used for Windows AI features so their models probably work great on Windows x Qualcomm. ONNX is great for smol models though, like VAD, a god send even.)