It's a crazy upside-down world where the Mac Studio M3 Ultra 512GB is the reasonable option among the alternatives if you intend to run larger models at usable(ish) speeds.
Experimenting with Local LLMs on macOS
91–100 of 276 posts
Re: Experimenting with Local LLMs on macOS
#92I agree that it's kind of magical that you can download a ~10GB file and suddenly your laptop is running something that can summarize text, answer questions and even reason a bit. The trick is balancing model size vs RAM: 12B–20B is about the upper limit for a 16GB machine without it choking. What I find interesting is that these models don't actually hit Apple's Neural Engine, they run on the GPU via Metal. Core ML…
I feel like Apple needs a new CEO, I've felt this way for a long time. If I had been in charge of Apple I would have embraced local LLMs and built an inference engine that optimizes models that are designed for Nvidia, I also would have probably toyed around with the idea of selling server-grade Apple Silicon processors and opening up the GPU spec so people can build against it. Seems like Apple tries to play it too safe. While Tim Cook is good as a COO, he's still running Apple as a COO. They need a man of vision, not a COO at the helm.
Re: Experimenting with Local LLMs on macOS
#93Earlier quoted context omitted.
Not sure about the Mac Pro, since you pay a lot for the big fancy case. The Studio seems more sensible. And of course Nvidia and AMD are coming out with options for massive amounts of high bandwidth GPU memory in desktop form factors. I like the idea of having basically a local LLM server that your laptop or other devices can connect to. Then your laptop doesn’t have to burn its battery on LLM work and it’s still loc…
> Not sure about the Mac Pro, since you pay a lot for the big fancy case. The Studio seems more sensible Oh wow, a maxed out Studio could run a 600B parameter model entirely in memory. Not bad for $12k. There may be a business in creating the software that links that box to an app on your phone.
That software is an HTTP request, no?
Any number of AI apps allow you to specify a custom endpoint. As long as your AI server accepts connections to the internet, you're gravy.
Re: Experimenting with Local LLMs on macOS
#94Oddly, my 2013 MacPro (Trashcan) runs LLMs pretty well, mostly because 64Gb of old school RAM is, like, $25.
Re: Experimenting with Local LLMs on macOS
#95+1 to LM Studio. Helped build a lot of intuition. Seeing and navigating all the configs helped me build intuition around what my macbook can or cannot do, how things are configured, how they work, etc... Great way to spend an hour or two.
I also like that it ships with some cli tools, including an openai compatible server. It’s great to be able to take a model that’s loaded and open up an endpoint to it for running local scripts.
You can get a quick feel for how it works via the chat interface and then extend it programmatically.
Re: Experimenting with Local LLMs on macOS
#96Earlier quoted context omitted.
Perhaps said software could even form an end to end encrypted tunnel from your phone to your local LLM server anywhere over the internet via a simple server intermediary. The amount of data transferred is tiny and the latency costs are typically going to be dominated by the LLM inference anyway. Not much advantage to doing LAN only except that you don’t need a server. Though the amount of people who care enough to bu…
> amount of people who care enough to buy a $3k - $10k server and set this up compared to just using ChatGPT is probably very small So I maxed that out, and it’s with Apple’s margins. I suspect you could do it for $5k. I’d also note that for heavy users of ChatGPT, the difference in energy costs for a home setup and the price for ChatGPT tokens may make this financially compelling for heavy users.
True, it may be profitable for pro users. At $200 a month for ChatGPT Pro, it may only take a few years to recoup the initial costs. Not sure about energy costs though.
And of course you’d be getting a worse model, since no open source model currently is as good as the best proprietary ones.
Though that gap should narrow as the open models improve and the proprietary ones seemingly plateau.
Re: Experimenting with Local LLMs on macOS
#97I agree that it's kind of magical that you can download a ~10GB file and suddenly your laptop is running something that can summarize text, answer questions and even reason a bit. The trick is balancing model size vs RAM: 12B–20B is about the upper limit for a 16GB machine without it choking. What I find interesting is that these models don't actually hit Apple's Neural Engine, they run on the GPU via Metal. Core ML…
I feel like Apple needs a new CEO, I've felt this way for a long time. If I had been in charge of Apple I would have embraced local LLMs and built an inference engine that optimizes models that are designed for Nvidia, I also would have probably toyed around with the idea of selling server-grade Apple Silicon processors and opening up the GPU spec so people can build against it. Seems like Apple tries to play it too…
Sounds like you’ve got a solid handle on things - go do it!
Re: Experimenting with Local LLMs on macOS
#98I agree that it's kind of magical that you can download a ~10GB file and suddenly your laptop is running something that can summarize text, answer questions and even reason a bit. The trick is balancing model size vs RAM: 12B–20B is about the upper limit for a 16GB machine without it choking. What I find interesting is that these models don't actually hit Apple's Neural Engine, they run on the GPU via Metal. Core ML…
I feel like Apple needs a new CEO, I've felt this way for a long time. If I had been in charge of Apple I would have embraced local LLMs and built an inference engine that optimizes models that are designed for Nvidia, I also would have probably toyed around with the idea of selling server-grade Apple Silicon processors and opening up the GPU spec so people can build against it. Seems like Apple tries to play it too…
They have local LLMs, apple foundation models: https://developer.apple.com/documentation/FoundationModels
Re: Experimenting with Local LLMs on macOS
#99I don't think we're anywhere close to running cutting-edge LLMs on our phones or laptops. What may be around the corner is running great models on a box at home. The AI lives at home. Your thin client talks to it, maybe runs a smaller AI on device to balance latency and quality. (This would be a natural extension for Apple to go into with its Mac Pro line. $10 to 20k for a home LLM device isn't ridiculous.)
> $10 to 20k for a home LLM device isn't ridiculous.
That price is ridiculous for most people. Silicon Valley payscales can afford that much, but see how few Apple Vision Pros got sold for far less.
Re: Experimenting with Local LLMs on macOS
#100Earlier quoted context omitted.
> At that point you are almost paying more than the datacenter does for inference hardware Of course. You and I don't have their economies of scale.
Then please excuse me for calling your one-man $10,000 inference device ridiculous.
I mean, not really? Yeah, I pay to go to the movies and sit in a theater that they let me buy a ticket for, but that doesn't mean people that want to set up a nice home theater are ridiculous, they just care more about controlling and customizing their experience.