It is kinda obvious, but maybe that's why it's not stated anywhere: each browser session will result in a download of 500 MB to ~1 GB, depending on your model selection. So, it's better to add a disclaimer if you end up using WebLLM in a customer-facing site.
WebLLM: high-performance in-browser LLM inference engine
21–30 of 32 posts
Re: WebLLM: high-performance in-browser LLM inference engine
#22[deleted]
Re: WebLLM: high-performance in-browser LLM inference engine
#23[flagged]
Re: WebLLM: high-performance in-browser LLM inference engine
#24[dead]
Re: WebLLM: high-performance in-browser LLM inference engine
#25[flagged]
Re: WebLLM: high-performance in-browser LLM inference engine
#26Project is de facto dead, used it for many years and had to rip it out 6 months ago, don't waste your time.
Yeah, both WebLLM and mlc-llm have been mostly abandoned since NVIDIA acquired OctoAI in late 2024.
Re: WebLLM: high-performance in-browser LLM inference engine
#27Earlier quoted context omitted.
You can enable WebGPU support in Google Chrome by turning on hardware acceleration and activating the WebGPU flag. It Works.
I tried it and the experience was not great. I first enabled chrome://flags/#force-enable-webgpu-interop which did nothing. Next, I enabled chrome://flags/#enable-unsafe-webgpu which made some WebGPU demos work, but they all use my CPU's integrated GPU, which is worse than using no GPU acceleration at all. Next, I found out that there is a powerPreference: 'high-performance' option. It can be enabled with chrome://fl…
Hey, Firefox Web GPU engineer here. We would love to hear about crashers on any platform! If you need help filing a Bugzilla bug, or want to email me so I can do it, please LMK.
Re: WebLLM: high-performance in-browser LLM inference engine
#28Earlier quoted context omitted.
What did you switch to?
There's, quietly, a llama.cpp WebGPU backend that works *great*. Some hacking required, it's unsupported, a side project for one of the lead maintainers and someone in school. Note of caution, llama.cpp isn't what it was, the grunt-level maintainers are left to their own devices. There's one key subsystem where things break regularly and the engineering is poor, and the "lead maintainer" is aggro and isn't really inv…
I seem to be out of the loop when it comes to llama.cpp? What happened? Can you please provide some more pointers to sources so one can verify your claims? Do you see alternatives to llama.cpp in whose future you have more trust?
Re: WebLLM: high-performance in-browser LLM inference engine
#29Oh nice, this actually works in Safari on my Mac! Prefill: 181.2 tok/s, Decode: 58.5 tok/s on my M4 Max -- not too impressive for 1B, but definitely impressive that it runs in my browser!
Re: WebLLM: high-performance in-browser LLM inference engine
#30Earlier quoted context omitted.
I tried it and the experience was not great. I first enabled chrome://flags/#force-enable-webgpu-interop which did nothing. Next, I enabled chrome://flags/#enable-unsafe-webgpu which made some WebGPU demos work, but they all use my CPU's integrated GPU, which is worse than using no GPU acceleration at all. Next, I found out that there is a powerPreference: 'high-performance' option. It can be enabled with chrome://fl…
Hey, Firefox Web GPU engineer here. We would love to hear about crashers on any platform! If you need help filing a Bugzilla bug, or want to email me so I can do it, please LMK.
Thanks, that is very kind of you. I have submitted bugs in the past and was enthusiastic for GPU support for over 15 years now, but I lost my faith.