Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Ollama is now powered by MLX on Apple Silicon in preview

ollama.com

Post reply on HN
Print view View original

231–240 of 384 posts

Re: Ollama is now powered by MLX on Apple Silicon in preview

#231

Post by podlp »

karimf wrote: post #50
babblingfish wrote: post #2

LLMs on device is the future. It's more secure and solves the problem of too much demand for inference compared to data center supply, it also would use less electricity. It's just a matter of getting the performance good enough. Most users don't need frontier model performance.

Depending on the use case, the future is already here. For example, last week I built a real-time voice AI running locally on iPhone 15. One use case is for people learning speaking english. The STT is quite good and the small LLM is enough for basic conversation. https://github.com/fikrikarim/volocal

That’s awesome! I’ve got a similar project for macOS/ iOS using the Apple Intelligence models and on-device STT Transcriber APIs. Do you think it the models you’re using could be quantized more that they could be downloaded on first run using Background Assets? Maybe we’re not there yet, but I’m interested in a better, local Siri like this with some sort of “agentic lite” capabilities.

Re: Ollama is now powered by MLX on Apple Silicon in preview

#232

Post by podlp »

adam_patarino wrote: post #204
babblingfish wrote: post #2

LLMs on device is the future. It's more secure and solves the problem of too much demand for inference compared to data center supply, it also would use less electricity. It's just a matter of getting the performance good enough. Most users don't need frontier model performance.

[flagged]

Rig sounds cool, I just joined the waitlist! I’m building something similar although with a much narrower purpose. Excited to learn more

Re: Ollama is now powered by MLX on Apple Silicon in preview

#233

Post by Aurornis »

babblingfish wrote: post #2

LLMs on device is the future. It's more secure and solves the problem of too much demand for inference compared to data center supply, it also would use less electricity. It's just a matter of getting the performance good enough. Most users don't need frontier model performance.

> solves the problem of too much demand for inference compared to data center supply

Maybe in the distant future when device compute capacity has increased by multiples and efficiency improvements have made smaller LLMs better.

The current data center buildouts are using GPU clusters and hybrid compute servers that are so much more powerful than anything you can run at home that they’re not in the same league. Even among the open models that you can run at home if you’re willing to spend $40K on hardware, the prefill and token generation speeds are so slow compared to SOTA served models that you really have to be dedicated to avoiding the cloud to run these.

We won’t be in a data center crunch forever. I would not be surprised if we have a period of data center oversupply after this rush to build out capacity.

However at the current rate of progress I don’t see local compute catching up to hosted models in quality and usability (speed) before data center capacity catches up to demand. This is coming from someone who spends more than is reasonable on local compute hardware.

Re: Ollama is now powered by MLX on Apple Silicon in preview

#234

Post by raw_anon_1111 »

abu_ameena wrote: post #223

On-device models are the future. Users prefer them. No privacy issues. No dealing with connectivity, tokens, or changes to vendors implementations. I have an app using Foundation Model, and it works great. I only wish I could backport it to pre macOS 26 versions.

Users don’t care about “privacy”. If they did, Meta and Alphabet wouldn’t be worth $1T+.

Users really don’t matter at all. The revenue for AI companies will be B2B where the user is not the customer - including coding agents. Most people don’t even use computers as their primary “computing device” and most people are buying crappy low end Android phones - no I’m not saying all Android phones are crappy. But that’s what most people are buying with the average selling price of an Android phone being $300.

Re: Ollama is now powered by MLX on Apple Silicon in preview

#235

Post by functional_dev »

domh wrote: post #165

I have an M4 Max with 48GB RAM. Anyone have any tips for good local models? Context length? Using the model recommended in the blog post (qwen3.5:35b-a3b-coding-nvfp4) with Ollama 0.19.0 and it can take anywhere between 6-25 seconds for a response (after lots of thinking) from me asking "Hello world". Is this the best that's currently achievable with my hardware or is there something that can be configured to get bet…

I did not know, that NVFP4 was handled at the silicon level... until I dug deeper here - https://vectree.io/c/llm-quantization-from-weights-to-bits-g...

Re: Ollama is now powered by MLX on Apple Silicon in preview

#237

Post by hamdingers »

robotswantdata wrote: post #83

Why are people still using Ollama? Serious. Lemonade or even llama.cpp are much better optimised and arguably just as easy to use.

Why not? Also serious.

It seems to just work every time I try to use it, the API is easy to work with, the model library is convenient. I've never hit any kind of snag that makes me look elsewhere.

Re: Ollama is now powered by MLX on Apple Silicon in preview

#238

Post by testing22321 »

abu_ameena wrote: post #223

On-device models are the future. Users prefer them. No privacy issues. No dealing with connectivity, tokens, or changes to vendors implementations. I have an app using Foundation Model, and it works great. I only wish I could backport it to pre macOS 26 versions.

I see all these LLM posts about if a certain model can run locally on certain hardware and I don’t get it.

What are you doing with these local models that run at x tokens/sec.

Do you have the equivalent of ChatGPT running entirely locally? What do you do with it? Why? I honestly don’t understand the point or use case.

Re: Ollama is now powered by MLX on Apple Silicon in preview

#239

Post by EagnaIonat »

braum wrote: post #236

How does Ollama help with Claude Code? Claude code runs in terminal but AFAIK connects back to anthropic directly and cannot run locally. I hope I'm missing something obvious.

You can create an MCP to call out to Ollama. Then have Claude farm work out to local models where the raw power isn't required. You can then have Claude review the work from the model.

Its not 100% offline, but there is a dramatic drop in token usage. As long as you can put up with the speed.

Re: Ollama is now powered by MLX on Apple Silicon in preview

#240

Post by EagnaIonat »

domh wrote: post #165

I have an M4 Max with 48GB RAM. Anyone have any tips for good local models? Context length? Using the model recommended in the blog post (qwen3.5:35b-a3b-coding-nvfp4) with Ollama 0.19.0 and it can take anywhere between 6-25 seconds for a response (after lots of thinking) from me asking "Hello world". Is this the best that's currently achievable with my hardware or is there something that can be configured to get bet…

When MLX comes out you will see a huge difference. I currently moved to LMStudio as it currently supports MLX.
Post reply on HN