Experimenting with Local LLMs on macOS
blog.6nok.org
Experimenting with Local LLMs on macOS
1–10 of 276 posts
Re: Experimenting with Local LLMs on macOS
#2Is anyone working on software that lets you run local LLMs in the browser?
In theory, it should be possible, shouldn't it?
The page could hold only the software in JavaScript that uses WebGL to run the neural net. And offer an "upload" button that the user can click to select a model from their file system. The button would not upload the model to a server - it would just let the JS code access it to convert it into WebGL and move it into the GPU.
This way, one could download models from HuggingFace, store them locally and use them as needed. Nicely sandboxed and independent of the operating system.
Re: Experimenting with Local LLMs on macOS
#3Is anyone working on software that lets you run local LLMs in the browser? In theory, it should be possible, shouldn't it? The page could hold only the software in JavaScript that uses WebGL to run the neural net. And offer an "upload" button that the user can click to select a model from their file system. The button would not upload the model to a server - it would just let the JS code access it to convert it into…
Yes. MLC's inference engine runs on WebGPU/WASM.
https://github.com/mlc-ai/web-llm-chat
Re: Experimenting with Local LLMs on macOS
#4Is anyone working on software that lets you run local LLMs in the browser? In theory, it should be possible, shouldn't it? The page could hold only the software in JavaScript that uses WebGL to run the neural net. And offer an "upload" button that the user can click to select a model from their file system. The button would not upload the model to a server - it would just let the JS code access it to convert it into…
Have you seen/used the webGPU spaces?
https://huggingface.co/docs/transformers.js/en/guides/webgpu
eta: its predecessor was using webGL
Re: Experimenting with Local LLMs on macOS
#5Is anyone working on software that lets you run local LLMs in the browser? In theory, it should be possible, shouldn't it? The page could hold only the software in JavaScript that uses WebGL to run the neural net. And offer an "upload" button that the user can click to select a model from their file system. The button would not upload the model to a server - it would just let the JS code access it to convert it into…
Is Open WebUI something like you are looking for? The design has some awkwardness, but overall it's incorporated a ton of great features.
Re: Experimenting with Local LLMs on macOS
#6Is anyone working on software that lets you run local LLMs in the browser? In theory, it should be possible, shouldn't it? The page could hold only the software in JavaScript that uses WebGL to run the neural net. And offer an "upload" button that the user can click to select a model from their file system. The button would not upload the model to a server - it would just let the JS code access it to convert it into…
You don’t need a browser to sandbox something. Easier and more performant to do GOU pass through to a container or VM.
Re: Experimenting with Local LLMs on macOS
#7Is anyone working on software that lets you run local LLMs in the browser? In theory, it should be possible, shouldn't it? The page could hold only the software in JavaScript that uses WebGL to run the neural net. And offer an "upload" button that the user can click to select a model from their file system. The button would not upload the model to a server - it would just let the JS code access it to convert it into…
Is Open WebUI something like you are looking for? The design has some awkwardness, but overall it's incorporated a ton of great features. https://openwebui.com/
No, I'm looking for an html page with a button "Select LLM". After pressing that button and selecting a local LLM from disk, it would show an input field where you can type your question and then it would use the given LLM to create the answer.
I'm not sure what OpenWebUI is, but if it was what I mean, they would surely have the page live and not ask users to install Docker etc.
Re: Experimenting with Local LLMs on macOS
#8Is anyone working on software that lets you run local LLMs in the browser? In theory, it should be possible, shouldn't it? The page could hold only the software in JavaScript that uses WebGL to run the neural net. And offer an "upload" button that the user can click to select a model from their file system. The button would not upload the model to a server - it would just let the JS code access it to convert it into…
It was done with gemma-3-270m, I hope someone will post a link to it below
Re: Experimenting with Local LLMs on macOS
#9Is anyone working on software that lets you run local LLMs in the browser? In theory, it should be possible, shouldn't it? The page could hold only the software in JavaScript that uses WebGL to run the neural net. And offer an "upload" button that the user can click to select a model from their file system. The button would not upload the model to a server - it would just let the JS code access it to convert it into…
Not browser but Electron. For the browser you would have to run a local nodejs server and point the browser app to use the local API. I use electron with nodejs and react for UI. Yes I can switch models.
Re: Experimenting with Local LLMs on macOS
#10Is anyone working on software that lets you run local LLMs in the browser? In theory, it should be possible, shouldn't it? The page could hold only the software in JavaScript that uses WebGL to run the neural net. And offer an "upload" button that the user can click to select a model from their file system. The button would not upload the model to a server - it would just let the JS code access it to convert it into…
Yes. MLC's inference engine runs on WebGPU/WASM. https://github.com/mlc-ai/web-llm-chat https://github.com/mlc-ai/mlc-llm https://github.com/mlc-ai/web-llm
Yeah, something like that, but without the WebGPU requirement.
Neither FireFox nor Chromium support WebGPU on Linux. Maybe behind flags. But before using a technology, I would wait until it is available in the default config.
Lets see when browsers will bring WebGPU to Linux.