Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Ollama is now powered by MLX on Apple Silicon in preview

ollama.com

Post reply on HN
Print view View original

371–380 of 384 posts

Re: Ollama is now powered by MLX on Apple Silicon in preview

#371

Post by zozbot234 »

LuxBennu wrote: post #370
yg1112 wrote: post #305

Earlier quoted context omitted.

The key difference is that MLX's array model assumes unified memory from the ground up. llama.cpp's Metal backend works fine but carries abstractions from the discrete GPU world — explicit buffer synchronization, command buffer boundaries — that are unnecessary when CPU and GPU share the same address space. You'll notice the gap most at large context lengths where KV cache pressure is highest.

that tracks with what i've noticed practically. shorter prompts feel basically the same between llama.cpp metal and what i'd expect from native mlx, but once context gets longer the overhead starts showing up. would be interesting to see if ollama's mlx path actually handles kv cache differently under the hood or if it just skips the buffer sync layer

If it's just about skipping some buffer sync that's something that could also be adopted by llama.cpp's own Metal backend, at least on Apple Silicon platforms.

Re: Ollama is now powered by MLX on Apple Silicon in preview

#372

Post by Aissen »

JambalayaJimbo wrote: post #263
raw_anon_1111 wrote: post #246

Earlier quoted context omitted.

70% of the world’s population use at least one Meta property at least once per day. How many of the other 30% are too poor/young/computer illiterate to be part of an addressable market? Every company has dozens of SaaS products that store their business critical information. Amazon installs Office on each computer, Slack (they were moving away from Chime when I left), and the sales department uses SalesForce - SA’s a…

The banking industry absolutely does care about privacy of their business data btw. We do use tools like Confluence but they're all hosted in our own data centers.

What's the plan to migrate off of it ? https://www.atlassian.com/licensing/data-center-end-of-life#...

Re: Ollama is now powered by MLX on Apple Silicon in preview

#373

Post by fennecfoxy »

robotswantdata wrote: post #83

Why are people still using Ollama? Serious. Lemonade or even llama.cpp are much better optimised and arguably just as easy to use.

I agree. In my experience on a macbook as well it can be terrifically hard to get some models to run properly on gpu w/ ollama, let alone containerised ollama.

Re: Ollama is now powered by MLX on Apple Silicon in preview

#374

Post by lxgr »

selcuka wrote: post #360
lxgr wrote: post #70

Earlier quoted context omitted.

Have you used GPT instant or mini yourself? I think it’s pretty cynical to assume that this is “good enough for most people”, even if they don’t know the difference between that and better models.

> I think it’s pretty cynical to assume that this is “good enough for most people” It's a deduction, not an assumption. Obviously it's "good enough" for "most people". Otherwise nobody would be using the free version of ChatGPT today. I pay for a Claude subscription, but even then I sometimes downgrade to Sonnet or even Haiku when I need a quick answer.

> Obviously it's "good enough" for "most people". Otherwise nobody would be using the free version of ChatGPT today.

I'd say it's better than nothing, which to me is not the same thing at all as "good enough".

For example, I believe most people would be better off with half the allowable queries per day, routed to a better model, but that's not an available product.

Re: Ollama is now powered by MLX on Apple Silicon in preview

#376

Post by xprnio »

charcircuit wrote: post #306
roadside_picnic wrote: post #285

Earlier quoted context omitted.

> Users don’t care about “privacy”. I worked for a research focused AI startup that had a strict "no external LLM" policy for code touching our core research. You're right that the average consumer doesn't care about privacy, but there are many, many users who do. The average consumer also don't have a desktop with GPU or high end Mac Studio, but that doesn't mean there aren't many people working with AI how do have…

Those users are addressed by being able to rent their own exclusive machines to run the model on. There will be some compromise that will be made to get access to the best intelligence available.

You will own nothing, and you will be greatful for it

Re: Ollama is now powered by MLX on Apple Silicon in preview

#377

Post by derangedHorse »

testing22321 wrote: post #238
abu_ameena wrote: post #223

On-device models are the future. Users prefer them. No privacy issues. No dealing with connectivity, tokens, or changes to vendors implementations. I have an app using Foundation Model, and it works great. I only wish I could backport it to pre macOS 26 versions.

I see all these LLM posts about if a certain model can run locally on certain hardware and I don’t get it. What are you doing with these local models that run at x tokens/sec. Do you have the equivalent of ChatGPT running entirely locally? What do you do with it? Why? I honestly don’t understand the point or use case.

Qwen3.5 is like an old version of ChatGPT and I can use it the same way I used GPT4 — writing emails, reading documentation and answering questions about it, reviewing code, answering trivia, etc.

Re: Ollama is now powered by MLX on Apple Silicon in preview

#378

Post by podlp »

karimf wrote: post #341
podlp wrote: post #231

Earlier quoted context omitted.

That’s awesome! I’ve got a similar project for macOS/ iOS using the Apple Intelligence models and on-device STT Transcriber APIs. Do you think it the models you’re using could be quantized more that they could be downloaded on first run using Background Assets? Maybe we’re not there yet, but I’m interested in a better, local Siri like this with some sort of “agentic lite” capabilities.

> Do you think it the models you’re using could be quantized more that they could be downloaded on first run using Background Assets? I first tried the Qwen 3.5 0.8B Q4_K_S and the model couldn't hold a basic conversation. Although I haven't tried lower quants on 2B. I'm also interested on the Apple Foundation models, and it's something I plan to try next. AFAIK it's on par with Qwen-3-4B [0]. The biggest upside as y…

Subjectively, AFM isn’t even close to Qwen. It’s one of the weakest models I’ve used. I’m not even sure how many people have Apple Intelligence enabled. But I agree, there must be a huge onboarding win long-term using (and adapting) a model that’s already optimized for your machine. I’ve learned how to navigate most of its shortcomings, but it’s not the most pleasant to work with.

Re: Ollama is now powered by MLX on Apple Silicon in preview

#379

Post by JSR_FDED »

bigyabai wrote: post #298
innagadadavida wrote: post #262

Earlier quoted context omitted.

These are all great statistics, but how do you explain ClawdBot explosion. Even in lower income countries like China. So much demand that Apple can’t keep up production of Mac Minis. Why aren’t these folks going towards cloud solutions? Is it cost or is there some consideration for having more control over their data?

> Why aren’t these folks going towards cloud solutions? They are. The majority aren't doing inference on a Mac Mini, but instead using it as a local host for cloud-based inference. You could have the same general experience on a $200 Chromebook or $300 Windows box.

Not if you want to use Messages to talk to it.

Re: Ollama is now powered by MLX on Apple Silicon in preview

#380

Post by bigyabai »

JSR_FDED wrote: post #379
bigyabai wrote: post #298

Earlier quoted context omitted.

> Why aren’t these folks going towards cloud solutions? They are. The majority aren't doing inference on a Mac Mini, but instead using it as a local host for cloud-based inference. You could have the same general experience on a $200 Chromebook or $300 Windows box.

Not if you want to use Messages to talk to it.

If you refuse to abandon iMessage for $300 in MSRP savings, you are beyond helping.
Post reply on HN