101–110 of 276 posts
Earlier quoted context omitted.
> Hopefully Apple optimizes Core ML to map transformer workloads to the ANE. If you want to convert models to run on the ANE there are tools provided: > Convert models from TensorFlow, PyTorch, and other libraries to Core ML. https://apple.github.io/coremltools/docs-guides/index.html
I thought Apple MLX can do that if you convert your model using it https://mlx-framework.org/
Earlier quoted context omitted.
The same way you use a cloud LLM.
I think the point was that for example for programming, people perceive state of the art LLMs as being net positive contributors, at least for mainstream programming languages and tasks, and I guess local LLMs aren't net positive contributors (i.e. an experienced programmer can build the same thing at least as fast when using an LLM).
I agree that it's kind of magical that you can download a ~10GB file and suddenly your laptop is running something that can summarize text, answer questions and even reason a bit. The trick is balancing model size vs RAM: 12B–20B is about the upper limit for a 16GB machine without it choking. What I find interesting is that these models don't actually hit Apple's Neural Engine, they run on the GPU via Metal. Core ML…
I feel like Apple needs a new CEO, I've felt this way for a long time. If I had been in charge of Apple I would have embraced local LLMs and built an inference engine that optimizes models that are designed for Nvidia, I also would have probably toyed around with the idea of selling server-grade Apple Silicon processors and opening up the GPU spec so people can build against it. Seems like Apple tries to play it too…
Hardware-wise though, I actually agree - Apple has dropped the ball so hard here that it's dumbfounding. They're the only TSMC customer that could realistically ship a comparable volume of chips as Nvidia, even without really impacting their smartphone business. They have hardware designers who can design GPUs from scratch, write proprietary graphics APIs and fine-tune for power efficiency. The only organizational roadblock that I can see is the executive vision, which has been pretty wishy-washy on AI for a while now. Apple wants to build a CoreML silo in a world where better products exist everywhere, it's a dead-end approach that should have died back in 2018.
Contextually it's weird too, I've seen tons of people defend Cook's relationship with Trump as "his duty to shareholders" and the like. But whenever you mention crypto mining or AI datacenter markets, people act like Apple is above selling products that people want. Future MBAs will be taught about this hubris once the shape of the total damages come into view.
Earlier quoted context omitted.
I feel like Apple needs a new CEO, I've felt this way for a long time. If I had been in charge of Apple I would have embraced local LLMs and built an inference engine that optimizes models that are designed for Nvidia, I also would have probably toyed around with the idea of selling server-grade Apple Silicon processors and opening up the GPU spec so people can build against it. Seems like Apple tries to play it too…
Software-wise, it makes sense: Nvidia has the IP lead, industry buy-in and supports the OSes everyone wants to use. Hardware-wise though, I actually agree - Apple has dropped the ball so hard here that it's dumbfounding. They're the only TSMC customer that could realistically ship a comparable volume of chips as Nvidia, even without really impacting their smartphone business. They have hardware designers who can desi…
People also want comfortable mattresses and high quality coffee machines. Should Apple make them too?
Apple not being in a particular industry is a perfectly valid choice, which is not remotely comparable to protecting their interests in the industries they are currently in. Selling datacenter-bound products is something Apple is not _remotely_ equipped for, and staffing up to do so at reasonable scale would not be a trivial task.
As for crypto mining... JFC.
I agree that it's kind of magical that you can download a ~10GB file and suddenly your laptop is running something that can summarize text, answer questions and even reason a bit. The trick is balancing model size vs RAM: 12B–20B is about the upper limit for a 16GB machine without it choking. What I find interesting is that these models don't actually hit Apple's Neural Engine, they run on the GPU via Metal. Core ML…
I feel like Apple needs a new CEO, I've felt this way for a long time. If I had been in charge of Apple I would have embraced local LLMs and built an inference engine that optimizes models that are designed for Nvidia, I also would have probably toyed around with the idea of selling server-grade Apple Silicon processors and opening up the GPU spec so people can build against it. Seems like Apple tries to play it too…
Earlier quoted context omitted.
Plenty of hobbies are significantly more expensive than that.
The rallying cry of money-wasters the world over. "At least it's not avgas!"
Earlier quoted context omitted.
> Hopefully Apple optimizes Core ML to map transformer workloads to the ANE. If you want to convert models to run on the ANE there are tools provided: > Convert models from TensorFlow, PyTorch, and other libraries to Core ML. https://apple.github.io/coremltools/docs-guides/index.html
It is less about conversion and more about extending ANE support for transformer-style models or giving developers more control. The issue is in targeting specific hardware blocks. When you convert with coremltools, Core ML takes over and doesn't provide fine-grained control - run on GPU, CPU or ANE. Also, ANE isn't really designed with transformers in mind, so most LLM inference defaults to GPU.
Look for Apple to add matmul acceleration into the GPU instead. Thats how to truly speed up local LLMs.
My main concern with running LLMs locally so far is that it absolutely kills your battery if you're constantly inferencing.