Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Experimenting with Local LLMs on macOS

blog.6nok.org

Post reply on HN
Print view View original

11–20 of 276 posts

Re: Experimenting with Local LLMs on macOS

#11

Post by mg »

coip wrote: post #4
mg wrote: post #2

Is anyone working on software that lets you run local LLMs in the browser? In theory, it should be possible, shouldn't it? The page could hold only the software in JavaScript that uses WebGL to run the neural net. And offer an "upload" button that the user can click to select a model from their file system. The button would not upload the model to a server - it would just let the JS code access it to convert it into…

Have you seen/used the webGPU spaces? https://huggingface.co/docs/transformers.js/en/guides/webgpu eta: its predecessor was using webGL

WebGPU is not yet available in the default config of Linux browsers, so WebGL would have been perfect :)

Re: Experimenting with Local LLMs on macOS

#12

Post by daoboy »

I'm running Hermes Mistral and the very first thing it did was start hallucinating.

I recently started an audio dream journal and want to keep it private. Set up whisper to transcribe the .wav file and dump it in an Obsidian folder.

The plan was to put a local llm step in to clean up the punctuation and paragraphs. I entered instructions to clean the transcript without changing or adding anything else.

Hermes responded by inventing an intereview with Sun Tzu about why he wrote the Art of War. When I stopped the process it apologized and advised it misunderstood when I talked about Sun Tzu. I never mentioned Sun Tzu or even provided a transcript. Just instructions.

We went around with this for a while before I could even get it to admit the mistake, and it refused to identify why it occurred in the first place.

Having to meticulously check for weird hallucinations will be far more time consuming than just doing the editing myself. This same logic applies to a lot of the areas I'd like to have a local llm for. Hopefully they'll get there soon.

Re: Experimenting with Local LLMs on macOS

#13

Post by SparkyMcUnicorn »

mg wrote: post #10
SparkyMcUnicorn wrote: post #3

Earlier quoted context omitted.

Yes. MLC's inference engine runs on WebGPU/WASM. https://github.com/mlc-ai/web-llm-chat https://github.com/mlc-ai/mlc-llm https://github.com/mlc-ai/web-llm

Yeah, something like that, but without the WebGPU requirement. Neither FireFox nor Chromium support WebGPU on Linux. Maybe behind flags. But before using a technology, I would wait until it is available in the default config. Lets see when browsers will bring WebGPU to Linux.

This should be what you're looking for. It doesn't utilize the GPU, but WebGL support is in the TODOs.

https://github.com/ngxson/wllama

https://huggingface.co/spaces/ngxson/wllama

Re: Experimenting with Local LLMs on macOS

#14

Post by bravetraveler »

mg wrote: post #7
samsolomon wrote: post #5

Earlier quoted context omitted.

Is Open WebUI something like you are looking for? The design has some awkwardness, but overall it's incorporated a ton of great features. https://openwebui.com/

No, I'm looking for an html page with a button "Select LLM". After pressing that button and selecting a local LLM from disk, it would show an input field where you can type your question and then it would use the given LLM to create the answer. I'm not sure what OpenWebUI is, but if it was what I mean, they would surely have the page live and not ask users to install Docker etc.

It's both what you want and not; the chat/question interface is as you describe, lack-of-installation is not. The LLM work is offloaded to other software, not the browser.

I would like to skip maintaining all this crap, though: I like your approach

Re: Experimenting with Local LLMs on macOS

#15

Post by coffeecoders »

I agree that it's kind of magical that you can download a ~10GB file and suddenly your laptop is running something that can summarize text, answer questions and even reason a bit.

The trick is balancing model size vs RAM: 12B–20B is about the upper limit for a 16GB machine without it choking.

What I find interesting is that these models don't actually hit Apple's Neural Engine, they run on the GPU via Metal. Core ML isn't great for custom runtimes and Apple hasn't given low-level developer access to the ANE afaik. And then there is memory bandwidth and dedicated SRAM issues. Hopefully Apple optimizes Core ML to map transformer workloads to the ANE.

Re: Experimenting with Local LLMs on macOS

#17

Post by Jemaclus »

mg wrote: post #7
samsolomon wrote: post #5

Earlier quoted context omitted.

Is Open WebUI something like you are looking for? The design has some awkwardness, but overall it's incorporated a ton of great features. https://openwebui.com/

No, I'm looking for an html page with a button "Select LLM". After pressing that button and selecting a local LLM from disk, it would show an input field where you can type your question and then it would use the given LLM to create the answer. I'm not sure what OpenWebUI is, but if it was what I mean, they would surely have the page live and not ask users to install Docker etc.

You should install it, because it's exactly what you just described.

Edit: From a UI perspective, it's exactly what you described. There's a dropdown where you select the LLM, and there's a ChatGPT-style chatbox. You just docker-up and go to town.

Maybe I don't understand the rest of the request, but I can't imagine a software where a webpage exists and it just magically has LLMs available in the browser with no installation?

Re: Experimenting with Local LLMs on macOS

#18

Post by simonh »

daoboy wrote: post #12

I'm running Hermes Mistral and the very first thing it did was start hallucinating. I recently started an audio dream journal and want to keep it private. Set up whisper to transcribe the .wav file and dump it in an Obsidian folder. The plan was to put a local llm step in to clean up the punctuation and paragraphs. I entered instructions to clean the transcript without changing or adding anything else. Hermes respond…

It’s often been assumed that accuracy and ‘correctness’ would be easy to implement on computers because they operate on logic, in some sense. It’s originality and creativity that would be hard, or impossible because it’s not logical. Science Fiction has been full of such assumptions. Yet here we are, the actual problem is inventing new heavy enough training sticks to beat our AIs out of constantly making stuff up and lying about it.

I suppose we shouldn’t be surprised in hindsight. We trained them on human communicative behaviour after all. Maybe using Reddit as a source wasn’t the smartest move. Reddit in, Reddit out.

Re: Experimenting with Local LLMs on macOS

#19

Post by vonneumannstan »

mg wrote: post #2

Is anyone working on software that lets you run local LLMs in the browser? In theory, it should be possible, shouldn't it? The page could hold only the software in JavaScript that uses WebGL to run the neural net. And offer an "upload" button that the user can click to select a model from their file system. The button would not upload the model to a server - it would just let the JS code access it to convert it into…

This one is pretty cool. Compile the gguf of an OSS LLM directly into an executable. Will open an interface in the browser to chat. Can also launch an OpenAI API style interface hosted locally.

Doesn't work quite as well on Windows due to the executable file size limit but seems great for Mac/Linux flavors.

https://github.com/Mozilla-Ocho/llamafile

Re: Experimenting with Local LLMs on macOS

#20

Post by 01HNNWZ0MV43FF »

adastra22 wrote: post #6
mg wrote: post #2

Is anyone working on software that lets you run local LLMs in the browser? In theory, it should be possible, shouldn't it? The page could hold only the software in JavaScript that uses WebGL to run the neural net. And offer an "upload" button that the user can click to select a model from their file system. The button would not upload the model to a server - it would just let the JS code access it to convert it into…

You don’t need a browser to sandbox something. Easier and more performant to do GOU pass through to a container or VM.

Container or VM is a bigger commitment. VMs need root and containers need Docker group and something like docker-compose or a shell script or something.

idk it's just like, do I want to run to the store and buy a 24-pack of water bottles, and stash them somewhere, or do I want to open the tap and have clean drinking water

Post reply on HN