Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Experimenting with Local LLMs on macOS

blog.6nok.org

Post reply on HN
Print view View original

141–150 of 276 posts

Re: Experimenting with Local LLMs on macOS

#141

Post by simonw »

mg wrote: post #10
SparkyMcUnicorn wrote: post #3

Earlier quoted context omitted.

Yes. MLC's inference engine runs on WebGPU/WASM. https://github.com/mlc-ai/web-llm-chat https://github.com/mlc-ai/mlc-llm https://github.com/mlc-ai/web-llm

Yeah, something like that, but without the WebGPU requirement. Neither FireFox nor Chromium support WebGPU on Linux. Maybe behind flags. But before using a technology, I would wait until it is available in the default config. Lets see when browsers will bring WebGPU to Linux.

Firefox Nightly on macOS now supports WebGPU, and the documentation says the Linux build supports it too.

Re: Experimenting with Local LLMs on macOS

#142

Post by Miraste »

mrexroad wrote: post #123
aurareturn wrote: post #107

Earlier quoted context omitted.

I think if Cook had vision, he could have started something called Apple Enterprise and sold Apple Silicon as a server and made AI chips. I agree he’s too conservative and has no product vision. Great manager though.

They did have Xserve back in the day. As great as Apple silicon is for running local llms along with being a general-purpose computing device, it’s not clear that Apple silicon have enough of a differentiating advantage over a rack of nvidia gpus to make it worthwhile in enterprise…

Strange to be saying this about Apple products, but its advantage is that it's way, way cheaper.

Re: Experimenting with Local LLMs on macOS

#143

Post by [deleted] »

daoboy wrote: post #12

I'm running Hermes Mistral and the very first thing it did was start hallucinating. I recently started an audio dream journal and want to keep it private. Set up whisper to transcribe the .wav file and dump it in an Obsidian folder. The plan was to put a local llm step in to clean up the punctuation and paragraphs. I entered instructions to clean the transcript without changing or adding anything else. Hermes respond…

[deleted]

Re: Experimenting with Local LLMs on macOS

#144

Post by elAhmo »

giancarlostoro wrote: post #92
coffeecoders wrote: post #15

I agree that it's kind of magical that you can download a ~10GB file and suddenly your laptop is running something that can summarize text, answer questions and even reason a bit. The trick is balancing model size vs RAM: 12B–20B is about the upper limit for a 16GB machine without it choking. What I find interesting is that these models don't actually hit Apple's Neural Engine, they run on the GPU via Metal. Core ML…

I feel like Apple needs a new CEO, I've felt this way for a long time. If I had been in charge of Apple I would have embraced local LLMs and built an inference engine that optimizes models that are designed for Nvidia, I also would have probably toyed around with the idea of selling server-grade Apple Silicon processors and opening up the GPU spec so people can build against it. Seems like Apple tries to play it too…

I think shareholders are fine with Tim Cook as a CEO.

Re: Experimenting with Local LLMs on macOS

#145

Post by jeffybefffy519 »

punitvthakkar wrote: post #27

So far I've not run into the kind of use cases that local LLMs can convincingly provide without making me feel like I'm using the first ever ChatGPT from 2022, in that they are limited and quite limiting. I am curious about what use cases the community has found that work for them. The example that one user has given in this thread about their local LLM inventing a Sun Tzu interview is exactly the kind of limitation…

Gemma3 is pretty useful on a long haul flight without internet

Re: Experimenting with Local LLMs on macOS

#146

Post by seanmcdirmid »

aurareturn wrote: post #107
giancarlostoro wrote: post #92

Earlier quoted context omitted.

I feel like Apple needs a new CEO, I've felt this way for a long time. If I had been in charge of Apple I would have embraced local LLMs and built an inference engine that optimizes models that are designed for Nvidia, I also would have probably toyed around with the idea of selling server-grade Apple Silicon processors and opening up the GPU spec so people can build against it. Seems like Apple tries to play it too…

I think if Cook had vision, he could have started something called Apple Enterprise and sold Apple Silicon as a server and made AI chips. I agree he’s too conservative and has no product vision. Great manager though.

I was pleasantly surprised Apple Silicon came out at all. Someone has their eye on long term vision at Apple at least, they just didn't do this on a whim.

Re: Experimenting with Local LLMs on macOS

#147

Post by jen20 »

bigyabai wrote: post #130
jen20 wrote: post #106

Earlier quoted context omitted.

> But whenever you mention crypto mining or AI datacenter markets, people act like Apple is above selling products that people want. People also want comfortable mattresses and high quality coffee machines. Should Apple make them too? Apple not being in a particular industry is a perfectly valid choice, which is not remotely comparable to protecting their interests in the industries they are currently in. Selling dat…

Apple is perfectly well equipped to sell datacenter products. They've done it in the past, even supporting Nvidia's compute drivers along the way. If they have the staff to design consumer-facing and developer-facing experiences, why wouldn't they address the datacenter? Money is money. 10 years ago people would have laughed at the notion of Nvidia abandoning the gaming market, now it's their most lucrative option. A…

> They've done it in the past, even supporting Nvidia's compute drivers along the way. If they have the staff to design consumer-facing and developer-facing experiences, why wouldn't they address the datacenter?

They did a horrific job of it before. The staff to design consumer facing experiences are busy doing exactly that. The developer facing experiences are very lean. The bandwidth simply isn't there to do DC products. Nor is the supply chain. Nor is the service supply chain. Etc, etc.

Re: Experimenting with Local LLMs on macOS

#149

Post by dghlsakjg »

JumpCrisscross wrote: post #115
dghlsakjg wrote: post #93

Earlier quoted context omitted.

That software is an HTTP request, no? Any number of AI apps allow you to specify a custom endpoint. As long as your AI server accepts connections to the internet, you're gravy.

> That software is an HTTP request, no? You and I could write it. Most folks couldn’t. If AI plateaus, this would be a good hill to have occupied.

My point is, what is there to build?

The person that is willing to buy that appliance is likely heavily overlapped with the person that is more than capable of pointing one of the dozens of existing apps at a custom domain.

Everyone else will continue to just use app based subscriptions.

Streaming platforms have plateaued (at best), but self hosted media appliances are still vanishingly rare.

Why would AI buck the trend that every other computing service has followed?

Re: Experimenting with Local LLMs on macOS

#150

Post by zargon »

simonw wrote: post #60
JumpCrisscross wrote: post #31

I don't think we're anywhere close to running cutting-edge LLMs on our phones or laptops. What may be around the corner is running great models on a box at home. The AI lives at home. Your thin client talks to it, maybe runs a smaller AI on device to balance latency and quality. (This would be a natural extension for Apple to go into with its Mac Pro line. $10 to 20k for a home LLM device isn't ridiculous.)

Right now you can run some of the best available open weight models on a 512GB Mac Studio, which retails for around $10,000. Here's Qwen3-Coder-480B-A35B-Instruct running at 24 tokens/second at 4bit: https://twitter.com/awnihannun/status/1947771502058672219 and Deep Seek V3 0324 in 4-bit at 20 toks/sec https://twitter.com/awnihannun/status/1904177084609827054 You can also string two 512GB Mac Studios together using M…

What these tweets about Apple silicon never show you: waiting 20+ minutes for it to ingest 32k context tokens. (Probably a lot longer for these big models.)
Post reply on HN