Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Experimenting with Local LLMs on macOS

blog.6nok.org

Post reply on HN
Print view View original

211–220 of 276 posts

Re: Experimenting with Local LLMs on macOS

#211

Post by punitvthakkar »

narrator wrote: post #88
punitvthakkar wrote: post #27

So far I've not run into the kind of use cases that local LLMs can convincingly provide without making me feel like I'm using the first ever ChatGPT from 2022, in that they are limited and quite limiting. I am curious about what use cases the community has found that work for them. The example that one user has given in this thread about their local LLM inventing a Sun Tzu interview is exactly the kind of limitation…

I have tried a lot of different LLMs and Gemma3:27b on a 48gb+ Macbook is probably the best for analyzing diaries and personal stuff you don't want to share with the cloud. The China models are comically bad with life advice. For example, I asked Deepseek to read my diaries and talk to me about my life goals and it told me in a very Confucian manner what the proper relationships in my life were for my stage of life a…

That is fascinating. One insight I read about LLMs is that they do represent a world-view of the people who train it, and hence the country that ships the dominant LLM technology can spread widely its world-view on others. Your experience seems to validate that insight.

Re: Experimenting with Local LLMs on macOS

#212

Post by punitvthakkar »

dxetech wrote: post #34
punitvthakkar wrote: post #27

So far I've not run into the kind of use cases that local LLMs can convincingly provide without making me feel like I'm using the first ever ChatGPT from 2022, in that they are limited and quite limiting. I am curious about what use cases the community has found that work for them. The example that one user has given in this thread about their local LLM inventing a Sun Tzu interview is exactly the kind of limitation…

There are situations where internet access is limited, or where there are frequent outages. An outdated LLM might be more useful than none at all. For example: my internet is out due to a severe storm, what safety precautions do I need to take?

Yes - emergency use cases make tons of sense.

Re: Experimenting with Local LLMs on macOS

#213

Post by punitvthakkar »

bityard wrote: post #82
punitvthakkar wrote: post #27

So far I've not run into the kind of use cases that local LLMs can convincingly provide without making me feel like I'm using the first ever ChatGPT from 2022, in that they are limited and quite limiting. I am curious about what use cases the community has found that work for them. The example that one user has given in this thread about their local LLM inventing a Sun Tzu interview is exactly the kind of limitation…

I use a local LLM for lots of little things that I used to use search engines for. Defining words, looking up unicode symbols for copy/paste, reminders on how to do X in bash or Python. Sometimes I use it as a starting point for high-level questions and curiosity and then move to actual human content or larger online models for more details and/or fact-checking if needed. If your computer is somewhat modern and has a…

All totally valid points and insights. This is great, thank you!

Re: Experimenting with Local LLMs on macOS

#214

Post by punitvthakkar »

luckydata wrote: post #30
punitvthakkar wrote: post #27

So far I've not run into the kind of use cases that local LLMs can convincingly provide without making me feel like I'm using the first ever ChatGPT from 2022, in that they are limited and quite limiting. I am curious about what use cases the community has found that work for them. The example that one user has given in this thread about their local LLM inventing a Sun Tzu interview is exactly the kind of limitation…

Local models can do embedding very well, which is useful for things like building a screenshot manager for example.

Whoa. I didn't think about using embeddings for screenshot management. How would I do this?

Re: Experimenting with Local LLMs on macOS

#215

Post by nxobject »

aurareturn wrote: post #107
giancarlostoro wrote: post #92

Earlier quoted context omitted.

I feel like Apple needs a new CEO, I've felt this way for a long time. If I had been in charge of Apple I would have embraced local LLMs and built an inference engine that optimizes models that are designed for Nvidia, I also would have probably toyed around with the idea of selling server-grade Apple Silicon processors and opening up the GPU spec so people can build against it. Seems like Apple tries to play it too…

I think if Cook had vision, he could have started something called Apple Enterprise and sold Apple Silicon as a server and made AI chips. I agree he’s too conservative and has no product vision. Great manager though.

I think that would spread Apple’s chip team too thinly between competing priorities - and require them to do E2E stuff they’d never be interested in doing. What’s always happened, even during Jobs, is that Apple would do something nice and backend-y, and then not be able to keep it up as they’d pour resources into some consumer product. (See: WebObjects, Xserve, Mac OS Server.)

Re: Experimenting with Local LLMs on macOS

#217

Post by nxobject »

bigyabai wrote: post #104
giancarlostoro wrote: post #92

Earlier quoted context omitted.

I feel like Apple needs a new CEO, I've felt this way for a long time. If I had been in charge of Apple I would have embraced local LLMs and built an inference engine that optimizes models that are designed for Nvidia, I also would have probably toyed around with the idea of selling server-grade Apple Silicon processors and opening up the GPU spec so people can build against it. Seems like Apple tries to play it too…

Software-wise, it makes sense: Nvidia has the IP lead, industry buy-in and supports the OSes everyone wants to use. Hardware-wise though, I actually agree - Apple has dropped the ball so hard here that it's dumbfounding. They're the only TSMC customer that could realistically ship a comparable volume of chips as Nvidia, even without really impacting their smartphone business. They have hardware designers who can desi…

> They have hardware designers who can design GPUs from scratch, write proprietary graphics APIs and fine-tune for power efficiency. The only organizational roadblock that I can see is the executive vision, which has been pretty wishy-washy on AI since for a while now.

The vision since Jobs has always been “build a great consumer product and own as much as you can while doing so”. That’s exactly how all of the design parameters of Ax/Mx series were determined and relentlessly optimized for - the fact that they have a highly competitive uarch was a salutary side-effect, but not a planned goal.

Re: Experimenting with Local LLMs on macOS

#218

Post by theshrike79 »

crazygringo wrote: post #58
punitvthakkar wrote: post #27

So far I've not run into the kind of use cases that local LLMs can convincingly provide without making me feel like I'm using the first ever ChatGPT from 2022, in that they are limited and quite limiting. I am curious about what use cases the community has found that work for them. The example that one user has given in this thread about their local LLM inventing a Sun Tzu interview is exactly the kind of limitation…

I see local LLM's being used mainly for automation as opposed to factual knowledge -- for classification, summarization, search, and things like grammar checking. So they need to be smart about your desired language(s) and all the everyday concepts we use in it (so they can understand the content of documents and messages), but they don't need any of the detailed factual knowledge around human history, programming la…

And local models are static, predictable and don't just go away when a new one comes out.

This makes them perfect for automation tasks.

Re: Experimenting with Local LLMs on macOS

#219

Post by nxobject »

utyop22 wrote: post #163
elAhmo wrote: post #144

Earlier quoted context omitted.

I think shareholders are fine with Tim Cook as a CEO.

I sometimes read posts on here and just laugh. Its easy to sit in the armchair and say "just be a visionary bro" when they forget Tim worked under Steve for awhile before his death - he has some sense and understanding of what it takes to get a great product out of the door. Nvidia is generating a lot of revenue, sure - but what is the downstream impact on its customers with the hardware? All they have right now is n…

Arguably, it’s why investors go in for Apple in the first place: Apple’s revenue fundamentally comes from consumer spending, whose prospects are relatively well understood by the average investor.

(I think it’s why big shareholders don’t get angry that Apple doesn’t splash their cash around: their core value proposition is focused in a dizzying tech market; take it or leave it. It’s very Warren Buffett.)

Re: Experimenting with Local LLMs on macOS

#220

Post by volemo »

dxetech wrote: post #34
punitvthakkar wrote: post #27

So far I've not run into the kind of use cases that local LLMs can convincingly provide without making me feel like I'm using the first ever ChatGPT from 2022, in that they are limited and quite limiting. I am curious about what use cases the community has found that work for them. The example that one user has given in this thread about their local LLM inventing a Sun Tzu interview is exactly the kind of limitation…

There are situations where internet access is limited, or where there are frequent outages. An outdated LLM might be more useful than none at all. For example: my internet is out due to a severe storm, what safety precautions do I need to take?

Surely not the ones you get from an LLM?
Post reply on HN