Good to see Ollama is catching up with the times for inference on Mac. MLX powered inference makes a big difference, especially on M5 as their graphs point out.
What really has been a game changer for my workflow is using https://omlx.ai/ that has SSD KV cold caching. No longer have to worry about a session falling out of memory and needing to prefill again. Combine that with the M5 Max prefill speed means more time is spend on generation than waiting for 50k+ content window to process.
Ollama is now powered by MLX on Apple Silicon in preview
61–70 of 384 posts
Re: Ollama is now powered by MLX on Apple Silicon in preview
#62Earlier quoted context omitted.
I agree. I can totally see in the future that open source LLMs will turn into paying a lumpsum for the model. Many will shut down. Some will turn into closed source labs. When VCs inevitably ask their AI labs to start making money or shut down, those free open source LLMS will cease to be free. Chinese AI labs have to release free open source models because they distill from OpenAI and Anthropic. They will always be…
> have to release free open source models because they distill from OpenAI and Anthropic They dont really have to though, they just need to be good enough and cheaper (even if distilled). That being said, it is true they are gaining a lot of visibility (specially Qwen) because of being open-source(weight). Hardware-wise they seem they will catch-up in 3-5 years (Nvidia is kind of irrelevant, what matters is the node)…
I highly doubt they can catch up in 3-5 years to Nvidia.
Chips take about 3 years to design. Do you think China will have Feymann-level AI systems in 3 years?
I think in 3 years, they'll have H200-equivalent at home.
Re: Ollama is now powered by MLX on Apple Silicon in preview
#63What is the difference between Ollama, llama.cpp, ggml and gguf?
Ollama is a user-friendly UI for LLM inference. It is powered by llama.cpp (or a fork of it) which is more power-user oriented and requires command-line wrangling. GGML is the math library behind llama.cpp and GGUF is the associated file format used for storing LLM weights.
i've found llama.cpp (as i understand it, ollama now uses their own version of this) to work much better in practice, faster and much more flexible.
Re: Ollama is now powered by MLX on Apple Silicon in preview
#64Earlier quoted context omitted.
Any citations? Because that was my impression, too. I want frontier model performance for my coding assistant, but "most users" could do with smaller/faster models. ChatGPT free falls back to GPT-5.2 Mini after a few interactions.
Frontier model has much better knowledge and they usually hallucinate less. It's not about the coding capabilities, it's about how much you can trust the model.
re: trust-
Have you tried the free version of ChatGPT? It is positively appalling. It’s like GPT 3.5 but prompted to write three times as much as necessary to seem useful. I wonder how many people have embarrassed themselves, lost their jobs, and been critically misinformed. All easy with state-of-the-art models but seemingly a guarantee with the bottom sub-slop tier.
Is the average person just talking to it about their day or something?
Re: Ollama is now powered by MLX on Apple Silicon in preview
#65Earlier quoted context omitted.
It isn't going to replace cloud LLMs since cloud LLMs will always be faster in throughput and smarter. Cloud and local LLMs will grow together, not replace each other. I'm not convinced that local LLMs use less electricity either. Per token at the same level of intelligence, cloud LLMs should run circles around local LLMs in efficiency. If it doesn't, what are we paying hundreds of billions of dollars for? I think lo…
[flagged]
Yea I get that there will always be demand for local waifus. I never said local LLMs won't be a thing. I even said it will be a huge thing. Just won't replace cloud.
Re: Ollama is now powered by MLX on Apple Silicon in preview
#66Earlier quoted context omitted.
It did take decades to catch and surpass US car makers right?
About 2.5 decades from the start of the JVs, but they did it. Semiconductors and jet turbines are really the last two tech trees that China has yet to master.
Which might they master first?
Re: Ollama is now powered by MLX on Apple Silicon in preview
#67LLMs on device is the future. It's more secure and solves the problem of too much demand for inference compared to data center supply, it also would use less electricity. It's just a matter of getting the performance good enough. Most users don't need frontier model performance.
It feels like you'll soon need a local llm to intermediate with the remote llm, like an ad blocker for browsers to stop them injecting ads or remind you not to send corporate IP out onto the Internet.
Re: Ollama is now powered by MLX on Apple Silicon in preview
#68Earlier quoted context omitted.
Depending on the use case, the future is already here. For example, last week I built a real-time voice AI running locally on iPhone 15. One use case is for people learning speaking english. The STT is quite good and the small LLM is enough for basic conversation. https://github.com/fikrikarim/volocal
Brilliant. Hope to see you in the App Store!
Oh thank you! I wasn’t sure if it was worth submitting to the app store since it was just a research preview, but I could do it if people want it.
Re: Ollama is now powered by MLX on Apple Silicon in preview
#69[dead]
Re: Ollama is now powered by MLX on Apple Silicon in preview
#70Earlier quoted context omitted.
"Most users don't need frontier model performance" unfortunately, this is not the case.
Any citations? Because that was my impression, too. I want frontier model performance for my coding assistant, but "most users" could do with smaller/faster models. ChatGPT free falls back to GPT-5.2 Mini after a few interactions.
Have you used GPT instant or mini yourself? I think it’s pretty cynical to assume that this is “good enough for most people”, even if they don’t know the difference between that and better models.