Show HN: I built a free in-browser Llama 3 chatbot powered by WebGPU

I spent the last few days building out a nicer ChatGPT-like interface to use Mistral 7B and Llama 3 fully within a browser (no deps and installs).

I’ve used the WebLLM project by MLC AI for a while to interact with LLMs in the browser when handling sensitive data but I found their UI quite lacking for serious use so I built a much better interface around WebLLM.

I’ve been using it as a therapist and coach. And it’s wonderful knowing that my personal information never leaves my local computer.

Should work on Desktop with Chrome or Edge. Other browsers are adding WebGPU support as well - see the Github for details on how you can get it to work on other browsers.

Note: after you send the first message, the model will be downloaded to your browser cache. That can take a while depending on the model and your internet connection. But on subsequent page loads, the model should be loaded from the IndexedDB cache so it should be much faster.

The project is open source (Apache 2.0) on Github. If you like it, I’d love contributions, particularly around making the first load faster.

Github: https://github.com/abi/secret-llama Demo: https://secretllama.com

Show HN: I built a free in-browser Llama 3 chatbot powered by WebGPU
github.com

Yes, it only starts the download after you send the first message so visiting the site won’t use up any space.

Approx sizes are listed in the GitHub README.

Models are stored in indexeddb and will be managed by the browser. Might get evicted.

Very cool! I wish there was chat history.

Also if you click the "New Chat" button while an answer is generating I think some of the output gets fed back into the model, it causes some weird output [0] but was kind of cool/fun. Here is a video of it as well [1], I almost think this should be some kind of special mode you can run. I'd be interested to know what the bug causes, is it just the existing output sent as input or a subset of it? It might be fun to watch a chat bot just randomly hallucinate, especially on a local model.

[0] https://cs.joshstrange.com/07kPLPPW

[1] https://cs.joshstrange.com/4sxvt1Mc

EDIT: Looks like calling `engine.resetChat()` while it's generating will do it, but I'm not sure why it errors after a while (maybe runs out of tokens for output? Not sure) but it would be cool to have this run until you stop it, automatically changing every 10-30 seconds or so.

I see you have Phi1.5-q4f16_1-1k - any chance you could add Phi-3?