Ollama is now powered by MLX on Apple Silicon in preview

Post by karimf » Tue, Mar 31, 2026, 8:56 PM UTC

podlp wrote: post #231

karimf wrote: post #50

Earlier quoted context omitted.

Depending on the use case, the future is already here. For example, last week I built a real-time voice AI running locally on iPhone 15. One use case is for people learning speaking english. The STT is quite good and the small LLM is enough for basic conversation. https://github.com/fikrikarim/volocal

That’s awesome! I’ve got a similar project for macOS/ iOS using the Apple Intelligence models and on-device STT Transcriber APIs. Do you think it the models you’re using could be quantized more that they could be downloaded on first run using Background Assets? Maybe we’re not there yet, but I’m interested in a better, local Siri like this with some sort of “agentic lite” capabilities.

Post by mirekrusin » Tue, Mar 31, 2026, 9:13 PM UTC

aurareturn wrote: post #84

mirekrusin wrote: post #55

Earlier quoted context omitted.

Local RTX 5090 is actually faster than A100/H100.

It's a $4,000 GPU with 32GB of VRAM and needs a 1,000 watt PSU. It's not realistic for the masses. If it has something like 80GB of VRAM, it'll cost $10k. The actual local LLM chip is Apple Silicon starting at the M5 generation with matmul acceleration in the GPU. You can run a good model using an M5 Max 128GB system. Good prompt processing and token generation speeds. Good enough for many things. Apple accidentally…

Post by esseph » Tue, Mar 31, 2026, 9:34 PM UTC

raw_anon_1111 wrote: post #276

esseph wrote: post #271

Earlier quoted context omitted.

Oh I'm sure they'll continue to have some cloud services, no doubt. But look at VMware for example, even after the insane price increases. Nutanix also seems to be doing quite well. I'm seeing a fair amount of on-prem bare metal k8s too.

Again - anecdotes is not data. We have data. That would be about as silly as me citing my own experience as proof that “everyone is moving to AWS” when I work for a company that is exclusively an AWS partner consulting company.

Post by jkl5xx » Tue, Mar 31, 2026, 9:38 PM UTC

svachalek wrote: post #280

testing22321 wrote: post #238

Earlier quoted context omitted.

I see all these LLM posts about if a certain model can run locally on certain hardware and I don’t get it. What are you doing with these local models that run at x tokens/sec. Do you have the equivalent of ChatGPT running entirely locally? What do you do with it? Why? I honestly don’t understand the point or use case.

1. There are small local models that have the capabilities of frontier models a year ago 2. They aren't harvesting your data for government files or training purposes 3. They won't be altered overnight to push advertising or a political agenda 4. They won't have their pricing raised at will 5. They won't disappear as soon as their host wants you to switch

Post by Patrick_Devine » Tue, Mar 31, 2026, 10:31 PM UTC

karimf wrote: post #341

podlp wrote: post #231

Earlier quoted context omitted.

That’s awesome! I’ve got a similar project for macOS/ iOS using the Apple Intelligence models and on-device STT Transcriber APIs. Do you think it the models you’re using could be quantized more that they could be downloaded on first run using Background Assets? Maybe we’re not there yet, but I’m interested in a better, local Siri like this with some sort of “agentic lite” capabilities.

> Do you think it the models you’re using could be quantized more that they could be downloaded on first run using Background Assets? I first tried the Qwen 3.5 0.8B Q4_K_S and the model couldn't hold a basic conversation. Although I haven't tried lower quants on 2B. I'm also interested on the Apple Foundation models, and it's something I plan to try next. AFAIK it's on par with Qwen-3-4B [0]. The biggest upside as y…