Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Ollama is now powered by MLX on Apple Silicon in preview

ollama.com

Post reply on HN
Print view View original

51–60 of 384 posts

Re: Ollama is now powered by MLX on Apple Silicon in preview

#51

Post by troad »

babblingfish wrote: post #2

LLMs on device is the future. It's more secure and solves the problem of too much demand for inference compared to data center supply, it also would use less electricity. It's just a matter of getting the performance good enough. Most users don't need frontier model performance.

I very recently installed llama.cpp on my consumer-grade M4 MBP, and I've been having loads of fun poking and prodding the local models. There's now a ChatGPT style interface baked into llama.cpp, which is very handy for quick experimentation. (I'm not entirely sure what Ollama would get me that llama.cpp doesn't, happy to hear suggestions!)

There are some surprisingly decent models that happily fit even into a mere 16 gigs of RAM. The recent Qwen 3.5 9B model is pretty good, though it did trip all over itself to avoid telling me what happened on Tiananmen Square in 1989. (But then I tried something called "Qwen3.5-9B-Uncensored-HauhauCS-Aggressive", which veers so hard the other way that it will happily write up a detailed plan for your upcoming invasion of Belgium, so I guess it all balances out?)

Re: Ollama is now powered by MLX on Apple Silicon in preview

#52

Post by mirekrusin »

pezgrande wrote: post #14
babblingfish wrote: post #2

LLMs on device is the future. It's more secure and solves the problem of too much demand for inference compared to data center supply, it also would use less electricity. It's just a matter of getting the performance good enough. Most users don't need frontier model performance.

You could argue that the only reason we have good open-weight models is because companies are trying to undermine the big dogs, and they are spending millions to make sure they dont get too far ahead. If the bubble pops then there wont be incentive to keep doing it.

You can have viable business model around open weight models where you offer fine tuning at a fee.

Re: Ollama is now powered by MLX on Apple Silicon in preview

#54

Post by overfeed »

babblingfish wrote: post #2

LLMs on device is the future. It's more secure and solves the problem of too much demand for inference compared to data center supply, it also would use less electricity. It's just a matter of getting the performance good enough. Most users don't need frontier model performance.

> It's just a matter of getting the performance good enough.

Who will pay for the ongoing development of (near-)SoTA local models? The good open-weight models are all developed by for-profit companies - you know how that story will end.

Re: Ollama is now powered by MLX on Apple Silicon in preview

#55

Post by mirekrusin »

aurareturn wrote: post #9
babblingfish wrote: post #2

LLMs on device is the future. It's more secure and solves the problem of too much demand for inference compared to data center supply, it also would use less electricity. It's just a matter of getting the performance good enough. Most users don't need frontier model performance.

It isn't going to replace cloud LLMs since cloud LLMs will always be faster in throughput and smarter. Cloud and local LLMs will grow together, not replace each other. I'm not convinced that local LLMs use less electricity either. Per token at the same level of intelligence, cloud LLMs should run circles around local LLMs in efficiency. If it doesn't, what are we paying hundreds of billions of dollars for? I think lo…

Local RTX 5090 is actually faster than A100/H100.

Re: Ollama is now powered by MLX on Apple Silicon in preview

#56

Post by nikanj »

babblingfish wrote: post #2

LLMs on device is the future. It's more secure and solves the problem of too much demand for inference compared to data center supply, it also would use less electricity. It's just a matter of getting the performance good enough. Most users don't need frontier model performance.

That also means sending every user a copy of the model that you spend billions training. The current model (running the models at the vendor side) makes it much easier to protect that investment

Re: Ollama is now powered by MLX on Apple Silicon in preview

#57

Post by jl6 »

babblingfish wrote: post #2

LLMs on device is the future. It's more secure and solves the problem of too much demand for inference compared to data center supply, it also would use less electricity. It's just a matter of getting the performance good enough. Most users don't need frontier model performance.

Not sure about the using less electricity part. With batching, it’s more efficient to serve multiple users simultaneously.

Re: Ollama is now powered by MLX on Apple Silicon in preview

#58

Post by Barbing »

karimf wrote: post #50
babblingfish wrote: post #2

LLMs on device is the future. It's more secure and solves the problem of too much demand for inference compared to data center supply, it also would use less electricity. It's just a matter of getting the performance good enough. Most users don't need frontier model performance.

Depending on the use case, the future is already here. For example, last week I built a real-time voice AI running locally on iPhone 15. One use case is for people learning speaking english. The STT is quite good and the small LLM is enough for basic conversation. https://github.com/fikrikarim/volocal

Brilliant. Hope to see you in the App Store!

Re: Ollama is now powered by MLX on Apple Silicon in preview

#59

Post by TeMPOraL »

jl6 wrote: post #57
babblingfish wrote: post #2

LLMs on device is the future. It's more secure and solves the problem of too much demand for inference compared to data center supply, it also would use less electricity. It's just a matter of getting the performance good enough. Most users don't need frontier model performance.

Not sure about the using less electricity part. With batching, it’s more efficient to serve multiple users simultaneously.

Indeed. Data centers have so many ways and reasons to be much more energy-efficient than local compute it's not even funny.

Re: Ollama is now powered by MLX on Apple Silicon in preview

#60

Post by aurareturn »

seanmcdirmid wrote: post #41
aurareturn wrote: post #33

Earlier quoted context omitted.

It did take decades to catch and surpass US car makers right?

About 2.5 decades from the start of the JVs, but they did it. Semiconductors and jet turbines are really the last two tech trees that China has yet to master.

Right. When I said "they'll always be behind", I meant in the next 5-10 years. They're gated by EUV tech. And once they have EUV tech, they need to scale up chip manufacturing.
Post reply on HN