[dead]
WebLLM: high-performance in-browser LLM inference engine
11–20 of 32 posts
Re: WebLLM: high-performance in-browser LLM inference engine
#12[dead]
Re: WebLLM: high-performance in-browser LLM inference engine
#13Bake me a cake
responds with
> Error: Cannot initialize runtime because of requested maxStorageBuffersPerShaderStage exceeds limit. requested=10, limit=9.
Re: WebLLM: high-performance in-browser LLM inference engine
#14Re: WebLLM: high-performance in-browser LLM inference engine
#15A WebX technology that actually involves browsers!
Re: WebLLM: high-performance in-browser LLM inference engine
#16You might like webml-kit https://npm.im/webml-kit
Re: WebLLM: high-performance in-browser LLM inference engine
#17This seems to be the demo: https://chat.webllm.ai/ I am getting: WebGPUNotAvailableError: WebGPU is not supported in your current environment, but it is necessary to run the WebLLM engine. On both, FireFox and Chromium on Linux.
You can enable WebGPU support in Google Chrome by turning on hardware acceleration and activating the WebGPU flag. It Works.
I tried it and the experience was not great. I first enabled chrome://flags/#force-enable-webgpu-interop which did nothing. Next, I enabled chrome://flags/#enable-unsafe-webgpu which made some WebGPU demos work, but they all use my CPU's integrated GPU, which is worse than using no GPU acceleration at all. Next, I found out that there is a powerPreference: 'high-performance' option. It can be enabled with chrome://flags/#enable-webgpu-developer-features and I can now request a GPU with 'high-performance' power preference, although there is no way to check whether this actually selected the correct GPU, and it is useless to do it like that anyway, since no website actually sets this (experimental) option. Fortunately, there is the flag chrome://flags/#force-high-performance-gpu which should solve this, but it is not available for my platform. I have a fairly standard RTX 3060, but apparently, the most common GPU (according to the steam hardware survey) is not supported.
Unfortunately, Firefox does not work any better. After setting the flag dom.webgpu.enabled to true in about:config, a few WebGPU examples (e.g. https://webgpu.github.io/webgpu-samples/?sample=helloTriangl...) work, but many other examples crash the browser. And of course, Firefox can't select the correct GPU either.
Re: WebLLM: high-performance in-browser LLM inference engine
#18I really enjoy this engine. I’ve used it for personal projects, but it hasn’t been updated since Gemma 2. I suggest using Transformers.js instead these days.
Re: WebLLM: high-performance in-browser LLM inference engine
#19Earlier quoted context omitted.
You can enable WebGPU support in Google Chrome by turning on hardware acceleration and activating the WebGPU flag. It Works.
I tried it and the experience was not great. I first enabled chrome://flags/#force-enable-webgpu-interop which did nothing. Next, I enabled chrome://flags/#enable-unsafe-webgpu which made some WebGPU demos work, but they all use my CPU's integrated GPU, which is worse than using no GPU acceleration at all. Next, I found out that there is a powerPreference: 'high-performance' option. It can be enabled with chrome://fl…
Pretty much the same here with Edge,#enable-unsafe-webgpu got me the emulated swiftshader adapter, but only #force-high-performance-gpu made the dGPU appear in the adapter list of https://webgpureport.org/.
Re: WebLLM: high-performance in-browser LLM inference engine
#20This seems to be the demo: https://chat.webllm.ai/ I am getting: WebGPUNotAvailableError: WebGPU is not supported in your current environment, but it is necessary to run the WebLLM engine. On both, FireFox and Chromium on Linux.
For firefox you may need to enable `dom.webgpu.enabled` in `about:config`
It's on by default on Windows iirc - it's considered a potential security risk on Linux by Mozilla, so they ship it but it's turned off and it's up to the user to decide.
I think the reasoning is just because of how varied graphics drivers/stacks are on Linux compared to Windows/OSX and the attack surface been larger.