Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Experimenting with Local LLMs on macOS

blog.6nok.org

Post reply on HN
Print view View original

261–270 of 276 posts

Re: Experimenting with Local LLMs on macOS

#261

Post by giancarlostoro »

packetlost wrote: post #249
alt227 wrote: post #239

Earlier quoted context omitted.

Apple silicon does not compete well in multicore spaces. People seem to think that because it can run single core things really well on a laptop, it can do anything. Servers regularly have 100-200 cpu cores maxing out of rapid fire threads. This is not what Apple silicon excels at. On top of that, it only performs so well on consumer devices because they control the hardware and OS and can tune both together. Creatin…

I know off the top of my head at least 3 places that would happily purchase a couple of XServers (one of which probably still has one) running MacOS Server. Linux isn't as hard of a requirement as you think.

Hell... I can think of loads of places running servers on WINDOWS (namely all of my employers, including F500 companies) I am not surprised that someone would run macOS as a server. At least MacOS is Unix based ;)

Re: Experimenting with Local LLMs on macOS

#262

Post by jasonvorhe »

__loam wrote: post #248
giancarlostoro wrote: post #92

Earlier quoted context omitted.

I feel like Apple needs a new CEO, I've felt this way for a long time. If I had been in charge of Apple I would have embraced local LLMs and built an inference engine that optimizes models that are designed for Nvidia, I also would have probably toyed around with the idea of selling server-grade Apple Silicon processors and opening up the GPU spec so people can build against it. Seems like Apple tries to play it too…

I'm glad Tim is the CEO instead of you.

Why? This is something that plays into all of Apple's supposed strengths: Privacy/no strict cloud dependency/on-device compute, hardware/software optimization while owning the stack and combine that with good UI/UX for a broad target audience without sacrificing too much for the power users. OP never said that local AI would be the only topic a new CEO should focus on.

Re: Experimenting with Local LLMs on macOS

#263

Post by saagarjha »

giancarlostoro wrote: post #234
saagarjha wrote: post #171

Earlier quoted context omitted.

One does not simply put a 5090 into an existing chip.

Not what I am suggesting. However, having trained a few different things on a modest M4 Pro chip (so not even their absolute most powerful chips mind you), and using it for local-first AI inference, I can see the value. A single server could serve an LLM for a small business and cost a lot less than running the same inference through a 5090 in terms of power usage. I could also see universities giving this type of co…

I think a 5090 will handily beat it on power usage.

Re: Experimenting with Local LLMs on macOS

#264

Post by renmillar »

rukuu001 wrote: post #186
ghilston wrote: post #173

Earlier quoted context omitted.

Any willingness to share this script? I've been working on some code to ingest things and summarize for them and I haven't gotten to email just yet.

Watch this space. It’s pretty scrappy code and needs a cleanup. It also does other random stuff relating to calendar entries that I want to be reminded to appropriately prepare for. But yes I’ll share, and I guess post an update in this thread?

I’d suggest the simple approach: run that script through Claude and have it extract just the email processing parts to create a clean CLI tool. This seems like exactly the type of refactoring task that LLMs are really good at.

Re: Experimenting with Local LLMs on macOS

#265

Post by DrAwdeOccarim »

jokoon wrote: post #242
DrAwdeOccarim wrote: post #159

Earlier quoted context omitted.

Minstral small 3.2 Q4_K_M and Gemma 3 12b 4 bit are amazing. I run both in LM Studio on a Macbook Pro M3 Pro with 36GB of RAM.

can I call it from the command line?

Yes. LM Studio acts like an OAi endpoint when you turn the server on.

Re: Experimenting with Local LLMs on macOS

#266

Post by all2 »

theshrike79 wrote: post #225
all2 wrote: post #116

Earlier quoted context omitted.

Note that Qwen3 and Deepseek are hobbled in Ollama; they cannot use tools as the tool portion of the system prompt is missing. Recommendation: use something else to run the model. Ollama is convenient, but insufficient for tool use for these models.

Could you give a recommendation that works instead of saying what doesn't work?

I would, but I haven't found a working solution.

Re: Experimenting with Local LLMs on macOS

#267

Post by adastra22 »

01HNNWZ0MV43FF wrote: post #20
adastra22 wrote: post #6

Earlier quoted context omitted.

You don’t need a browser to sandbox something. Easier and more performant to do GOU pass through to a container or VM.

Container or VM is a bigger commitment. VMs need root and containers need Docker group and something like docker-compose or a shell script or something. idk it's just like, do I want to run to the store and buy a 24-pack of water bottles, and stash them somewhere, or do I want to open the tap and have clean drinking water

Neither of requirements are true on recent OS versions. Users have had the ability to make containers or VMs without special privileges for a very long time now.

Re: Experimenting with Local LLMs on macOS

#268

Post by j45 »

brookst wrote: post #232
jbverschoor wrote: post #124

Earlier quoted context omitted.

Local llm.. everybody is scared of privacy.. many people don’t want to buy subscriptions (still). Just sell a proper HomePod with 64GB-128GB ram, which handles everything including your personal LLM, Time Machine if needed, back to Mac (Tailscale/zerotier) + they can compete efficiently with the other. Cloud providers.

It’s a mistake to generalize from the HN population. Most people don’t care about privacy (see: success of Facebook and TikTok). Most people don’t care about subscriptions (see: cable TV, Netflix). There may be a niche market for a local inference device that costs $1000 and has to be replaced every year or two during the early days of AI, but it’s not a market with decent ROI for Apple.

An iPhone, Macbook, etc all cost in the $1000 range.

There was a post about the new iphone using A19, which includes a feature that makes local inference much easier.

If that makes it to M5, I think the local inference case continues to grow with each M processor.

Re: Experimenting with Local LLMs on macOS

#269

Post by j45 »

spease wrote: post #191
elAhmo wrote: post #144

Earlier quoted context omitted.

I think shareholders are fine with Tim Cook as a CEO.

Yes. And everyone is glossing over the benefit of unified memory for LLM applications. Apple may not have the models, but it has customer goodwill, a platform, and the logistical infrastructure to roll them out. It probably even has the cash to buy some AI companies outright; maybe not the big ones (for a reasonable amount, anyway) but small to midsize ones with domain-specific models that could be combined. Not to m…

Unified memory and examples like the M1 Ultra still being able to hold it's own years later might be one of the things that not all Mac users and non-mac users alike have experienced.

It's nice to see 16 Gb becoming the minimum, to me it should have been 32 for a long time.

Re: Experimenting with Local LLMs on macOS

#270

Post by wpm »

nxobject wrote: post #215
aurareturn wrote: post #107

Earlier quoted context omitted.

I think if Cook had vision, he could have started something called Apple Enterprise and sold Apple Silicon as a server and made AI chips. I agree he’s too conservative and has no product vision. Great manager though.

I think that would spread Apple’s chip team too thinly between competing priorities - and require them to do E2E stuff they’d never be interested in doing. What’s always happened, even during Jobs, is that Apple would do something nice and backend-y, and then not be able to keep it up as they’d pour resources into some consumer product. (See: WebObjects, Xserve, Mac OS Server.)

WebObjects was from NeXT. The Xserve saw regular updates until 2010. Mac OS Server was a GUI for a bunch of open source tools, and where it wasn’t (Workgroup Manager), they replaced with the MDM.

Apple had more money when they killed these than they did when these products were introduced. It’s not a resources issue. It’s a care issue. Same reason they fired the Mac OS Automation team. Same reason their documentation sucks hot diarrhea and all their good stuff is in the “documentation archive”. Penny pinching. “Shareholder value”. New blood destroying shit they didn’t understand.

Post reply on HN