Experimenting with Local LLMs on macOS • Hacker News Board

Re: Experimenting with Local LLMs on macOS

#161

Post by jasonjmcghee » Mon, Sep 08, 2025, 11:26 PM UTC

bigyabai wrote: post #50

hu3 wrote: post #48

Earlier quoted context omitted.

I just tested Qwen3 A3B vs ChatGPE a random prompt from my head and: > Please write a C# middleware to block requests from browser agents that contain any word in a specified list of words: openai, grok, gemini, claude. I used ChatpGPT 4o from GitHub Copilot inside VSCode. And Qwen3 A3B from here: https://deepinfra.com/Qwen/Qwen3-30B-A3B ChatGPT 4o was considerably better. Less verbose and less unnecessary abstractio…

You want the 2507 update of the model, I think the one you used is ~8-10 months out-of-date.

No they want Qwen3-Coder-30B-A3B-Instruct

Re: Experimenting with Local LLMs on macOS

#162

Post by rukuu001 » Mon, Sep 08, 2025, 11:29 PM UTC

punitvthakkar wrote: post #27

So far I've not run into the kind of use cases that local LLMs can convincingly provide without making me feel like I'm using the first ever ChatGPT from 2022, in that they are limited and quite limiting. I am curious about what use cases the community has found that work for them. The example that one user has given in this thread about their local LLM inventing a Sun Tzu interview is exactly the kind of limitation…

I'm running Gemma3-270M locally (MLX). I got a Python script that pulls down emails based on a whitelist and summarises them. The 270M model does a good job of this. This is running in a terminal. It means I barely look at my email during the day.

Re: Experimenting with Local LLMs on macOS

#163

Post by utyop22 » Mon, Sep 08, 2025, 11:44 PM UTC

elAhmo wrote: post #144

giancarlostoro wrote: post #92

Earlier quoted context omitted.

I feel like Apple needs a new CEO, I've felt this way for a long time. If I had been in charge of Apple I would have embraced local LLMs and built an inference engine that optimizes models that are designed for Nvidia, I also would have probably toyed around with the idea of selling server-grade Apple Silicon processors and opening up the GPU spec so people can build against it. Seems like Apple tries to play it too…

I think shareholders are fine with Tim Cook as a CEO.

I sometimes read posts on here and just laugh.

Its easy to sit in the armchair and say "just be a visionary bro" when they forget Tim worked under Steve for awhile before his death - he has some sense and understanding of what it takes to get a great product out of the door.

Nvidia is generating a lot of revenue, sure - but what is the downstream impact on its customers with the hardware? All they have right now is negative returns to show for their spending. Could this change? Maybe. Is it likely? Not in my view.

As it stands, Apple has made the absolute right choice in not wasting its cash and is demonstrating discipline. Which when all this LLM mania quietens, shareholders will respect.

Re: Experimenting with Local LLMs on macOS

#164

Post by alexashka » Mon, Sep 08, 2025, 11:49 PM UTC

aurareturn wrote: post #107

giancarlostoro wrote: post #92

Earlier quoted context omitted.

I feel like Apple needs a new CEO, I've felt this way for a long time. If I had been in charge of Apple I would have embraced local LLMs and built an inference engine that optimizes models that are designed for Nvidia, I also would have probably toyed around with the idea of selling server-grade Apple Silicon processors and opening up the GPU spec so people can build against it. Seems like Apple tries to play it too…

I think if Cook had vision, he could have started something called Apple Enterprise and sold Apple Silicon as a server and made AI chips. I agree he’s too conservative and has no product vision. Great manager though.

[flagged]

Re: Experimenting with Local LLMs on macOS

#165

Post by xanderlewis » Tue, Sep 09, 2025, 12:10 AM UTC

alexashka wrote: post #164

aurareturn wrote: post #107

Earlier quoted context omitted.

I think if Cook had vision, he could have started something called Apple Enterprise and sold Apple Silicon as a server and made AI chips. I agree he’s too conservative and has no product vision. Great manager though.

[flagged]

Anyone who doesn’t happen to do exactly what I do and have the same interests as me is ‘80 IQ’ — whatever that means. Got it.

Re: Experimenting with Local LLMs on macOS

#166

Post by elpakal » Tue, Sep 09, 2025, 12:27 AM UTC

woadwarrior01 wrote: post #101

ls-a wrote: post #62

Earlier quoted context omitted.

I thought Apple MLX can do that if you convert your model using it https://mlx-framework.org/

MLX does not support the ANE. https://github.com/ml-explore/mlx/issues/18

Yes it does.

That’s just an issue with stale and incorrect information. Here are the docs https://opensource.apple.com/projects/mlx/

Re: Experimenting with Local LLMs on macOS

#167

Post by colecut » Tue, Sep 09, 2025, 12:30 AM UTC

tpae wrote: post #103

Check out Osaurus - MIT Licensed, native, Apple Silicon–only local LLM server - https://github.com/dinoki-ai/osaurus

thank you

Re: Experimenting with Local LLMs on macOS

#168

Post by milgrum » Tue, Sep 09, 2025, 12:34 AM UTC

data-ottawa wrote: post #85

JumpCrisscross wrote: post #31

I don't think we're anywhere close to running cutting-edge LLMs on our phones or laptops. What may be around the corner is running great models on a box at home. The AI lives at home. Your thin client talks to it, maybe runs a smaller AI on device to balance latency and quality. (This would be a natural extension for Apple to go into with its Mac Pro line. $10 to 20k for a home LLM device isn't ridiculous.)

This is what I’m doing with my amd 395+. I’m running docker containers with different apps and it works well enough for a lot of my use cases. I mostly use Qwen Code and GPT OSS 120b right now. When the next generation of this tech comes through I will probably upgrade despite the price, the value is worth it to me.

How many TPS do you get running GPT OSS 120b on the 395+? Considering a Framework desktop for a similar use case, but I’ve been reading mixed things about performance (specifically with regards to memory bandwidth, but I’m not sure if that’s really the underlying issue)

Re: Experimenting with Local LLMs on macOS

#169

Post by saagarjha » Tue, Sep 09, 2025, 12:47 AM UTC

alexashka wrote: post #164

aurareturn wrote: post #107

Earlier quoted context omitted.

I think if Cook had vision, he could have started something called Apple Enterprise and sold Apple Silicon as a server and made AI chips. I agree he’s too conservative and has no product vision. Great manager though.

[flagged]

By calling everyone who buys Apple products 80 IQ, you are lowering the quality of the discourse here. Please don't do that.

Re: Experimenting with Local LLMs on macOS

#170

Post by saagarjha » Tue, Sep 09, 2025, 12:49 AM UTC

bigyabai wrote: post #130

jen20 wrote: post #106

Earlier quoted context omitted.

> But whenever you mention crypto mining or AI datacenter markets, people act like Apple is above selling products that people want. People also want comfortable mattresses and high quality coffee machines. Should Apple make them too? Apple not being in a particular industry is a perfectly valid choice, which is not remotely comparable to protecting their interests in the industries they are currently in. Selling dat…

Apple is perfectly well equipped to sell datacenter products. They've done it in the past, even supporting Nvidia's compute drivers along the way. If they have the staff to design consumer-facing and developer-facing experiences, why wouldn't they address the datacenter? Money is money. 10 years ago people would have laughed at the notion of Nvidia abandoning the gaming market, now it's their most lucrative option. A…

Apple makes more profit on iPhones than Nvidia does on its entire datacenter business. Why would they want to enter a highly competitive market that they have no expertise in on a whim?