Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Experimenting with Local LLMs on macOS

blog.6nok.org

Post reply on HN
Print view View original

151–160 of 276 posts

Re: Experimenting with Local LLMs on macOS

#151

Post by codazoda »

segmondy wrote: post #102
oblio wrote: post #39

Earlier quoted context omitted.

I think the point was that for example for programming, people perceive state of the art LLMs as being net positive contributors, at least for mainstream programming languages and tasks, and I guess local LLMs aren't net positive contributors (i.e. an experienced programmer can build the same thing at least as fast when using an LLM).

I know this is false, DeepSeekv3.1, GLM4.5, KimiK2-0905, Qwen-235B are all solid open models. Last night, I vibed rough 1300 lines of C server code in about an hour. 0 compilation error, ran without errors and got the job done. I want to meet this experienced programmer that can knock out 1300 lines of C code in an hour.

How much machine do you have to be able to run Qwen-235B locally?

Re: Experimenting with Local LLMs on macOS

#152

Post by JumpCrisscross »

dghlsakjg wrote: post #149
JumpCrisscross wrote: post #115

Earlier quoted context omitted.

> That software is an HTTP request, no? You and I could write it. Most folks couldn’t. If AI plateaus, this would be a good hill to have occupied.

My point is, what is there to build? The person that is willing to buy that appliance is likely heavily overlapped with the person that is more than capable of pointing one of the dozens of existing apps at a custom domain. Everyone else will continue to just use app based subscriptions. Streaming platforms have plateaued (at best), but self hosted media appliances are still vanishingly rare. Why would AI buck the tr…

> what is there to build?

Integrated solution. You buy the box. You download the app. It works like the ChatGPT app, except it's tunneling to the box you have at home which has been preconfigured to work with the app. Maybe you have a subscription to keep everything up to date. Maybe you have an open-source model 'store'.

Re: Experimenting with Local LLMs on macOS

#153

Post by solardev »

narrator wrote: post #88
punitvthakkar wrote: post #27

So far I've not run into the kind of use cases that local LLMs can convincingly provide without making me feel like I'm using the first ever ChatGPT from 2022, in that they are limited and quite limiting. I am curious about what use cases the community has found that work for them. The example that one user has given in this thread about their local LLM inventing a Sun Tzu interview is exactly the kind of limitation…

I have tried a lot of different LLMs and Gemma3:27b on a 48gb+ Macbook is probably the best for analyzing diaries and personal stuff you don't want to share with the cloud. The China models are comically bad with life advice. For example, I asked Deepseek to read my diaries and talk to me about my life goals and it told me in a very Confucian manner what the proper relationships in my life were for my stage of life a…

Lol, that's actually kinda cool. Did you get any interesting Eastern responses to your diary entries?

I'm imagining something like...

> Dear diary, I got bullied again today, and the bread was stale in my PB&J :(

>> My son, remember this: The one who mocks others wounds his own virtue. The one who suffers mockery must guard his heart. To endure without hatred is strength; to strike without cause is disgrace. The noble one corrects himself first, then the world will follow.

Re: Experimenting with Local LLMs on macOS

#154

Post by drusepth »

segmondy wrote: post #102
oblio wrote: post #39

Earlier quoted context omitted.

I think the point was that for example for programming, people perceive state of the art LLMs as being net positive contributors, at least for mainstream programming languages and tasks, and I guess local LLMs aren't net positive contributors (i.e. an experienced programmer can build the same thing at least as fast when using an LLM).

I know this is false, DeepSeekv3.1, GLM4.5, KimiK2-0905, Qwen-235B are all solid open models. Last night, I vibed rough 1300 lines of C server code in about an hour. 0 compilation error, ran without errors and got the job done. I want to meet this experienced programmer that can knock out 1300 lines of C code in an hour.

Are 235B models classified as local LLMs? I guess they probably are, but others in this thread are probably looking more toward 20B-30B models and sizes that generally fit on the RAM you'd expect in average or slightly-higher-end hardware.

My beefy 3D gamedev workstation with a 4090 and 128GB RAM can't even run a 235B model unless it's extremely quantized (and even then, only at like single-digit tokens/minute).

Re: Experimenting with Local LLMs on macOS

#155

Post by simonw »

JumpCrisscross wrote: post #53
brokencode wrote: post #47

Earlier quoted context omitted.

Not sure about the Mac Pro, since you pay a lot for the big fancy case. The Studio seems more sensible. And of course Nvidia and AMD are coming out with options for massive amounts of high bandwidth GPU memory in desktop form factors. I like the idea of having basically a local LLM server that your laptop or other devices can connect to. Then your laptop doesn’t have to burn its battery on LLM work and it’s still loc…

> Not sure about the Mac Pro, since you pay a lot for the big fancy case. The Studio seems more sensible Oh wow, a maxed out Studio could run a 600B parameter model entirely in memory. Not bad for $12k. There may be a business in creating the software that links that box to an app on your phone.

I have been using a Tailscale VPN to make LM Studio and Ollama running on my Mac available to my iPhone when I leave the house.

Re: Experimenting with Local LLMs on macOS

#156

Post by vonneumannstan »

JumpCrisscross wrote: post #31

I don't think we're anywhere close to running cutting-edge LLMs on our phones or laptops. What may be around the corner is running great models on a box at home. The AI lives at home. Your thin client talks to it, maybe runs a smaller AI on device to balance latency and quality. (This would be a natural extension for Apple to go into with its Mac Pro line. $10 to 20k for a home LLM device isn't ridiculous.)

Doesnt gpt-oss-120b perform better across the board at a fraction of the memory? Just specced a $4k mac studio that can easily run that at 128 gb memory.

Re: Experimenting with Local LLMs on macOS

#157

Post by vonneumannstan »

bigyabai wrote: post #37
JumpCrisscross wrote: post #31

I don't think we're anywhere close to running cutting-edge LLMs on our phones or laptops. What may be around the corner is running great models on a box at home. The AI lives at home. Your thin client talks to it, maybe runs a smaller AI on device to balance latency and quality. (This would be a natural extension for Apple to go into with its Mac Pro line. $10 to 20k for a home LLM device isn't ridiculous.)

> $10 to 20k for a home LLM device isn't ridiculous. At that point you are almost paying more than the datacenter does for inference hardware.

Almost? Isn't a single h100 like 30k which is the bare minimum to run a big model?

Re: Experimenting with Local LLMs on macOS

#158

Post by zackmorris »

coffeecoders wrote: post #15

I agree that it's kind of magical that you can download a ~10GB file and suddenly your laptop is running something that can summarize text, answer questions and even reason a bit. The trick is balancing model size vs RAM: 12B–20B is about the upper limit for a 16GB machine without it choking. What I find interesting is that these models don't actually hit Apple's Neural Engine, they run on the GPU via Metal. Core ML…

Don't get me started. Many new computers come with an NPU of some kind, which is superfluous to a GPU.

But what's really going on is that we never got the highly multicore and distributed computers that could have started going mainstream in the 1980s, and certainly by the late 1990s when high-speed internet hit. So single-threaded performance is about the same now as 20 years ago. Meanwhile video cards have gotten exponentially more powerful and affordable, but without the virtual memory and virtualization capabilities of CPUs, so we're seeing ridiculous artificial limitations like not being able to run certain LLMs because the hardware "isn't powerful enough", rather than just having a slower experience or borrowing the PC in the next room for more computing power.

To go to the incredible lengths that Apple went to in designing the M1, not just wrt hardware but in adding yet another layer of software emulation since the 68000 days, without actually bringing multicore with local memories to the level that today's VLSI design rules could allow, is laughable for me. If it wasn't so tragic.

It's hard for me to live and work in a tech status quo so far removed from what I had envisioned growing up. We're practically at AGI, but also mired in ensh@ttification. Reflected in politics too. We'll have the first trillionaire before we solve world hunger, and I'm bracing for Skynet/Ultron before we have C3P0/JARVIS.

Re: Experimenting with Local LLMs on macOS

#160

Post by DrAwdeOccarim »

jftuga wrote: post #71

I have a macbook air M4 with 32 GB. What LM Studio models would you recommend for: * General Q&A * Specific to programming - mostly Python and Go. I forgot the command now, but I did run a command that allowed MacOS to allocate and use maybe 28 GB of RAM to the GPU for use with LLMs.

I adore Qwen 3 30b a3b 2507. Pretty easy to write an MCP to let us search the web with Brave API key. I run it on my Macbook Pro M3 Pro 36 GB.
Post reply on HN