Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Ollama is now powered by MLX on Apple Silicon in preview

ollama.com

Post reply on HN
Print view View original

71–80 of 384 posts

Re: Ollama is now powered by MLX on Apple Silicon in preview

#71

Post by whackernews »

troad wrote: post #51
babblingfish wrote: post #2

LLMs on device is the future. It's more secure and solves the problem of too much demand for inference compared to data center supply, it also would use less electricity. It's just a matter of getting the performance good enough. Most users don't need frontier model performance.

I very recently installed llama.cpp on my consumer-grade M4 MBP, and I've been having loads of fun poking and prodding the local models. There's now a ChatGPT style interface baked into llama.cpp, which is very handy for quick experimentation. (I'm not entirely sure what Ollama would get me that llama.cpp doesn't, happy to hear suggestions!) There are some surprisingly decent models that happily fit even into a mere…

Oh does llama.cpp use MLX or whatever? I had this question, wonder if you know? A search suggests it doesn’t but I don’t really understand.

Re: Ollama is now powered by MLX on Apple Silicon in preview

#73

Post by universa1 »

charlotte12345 wrote: post #72

[flagged]

i am curious: is the performance gap between x86 cpu inference and apple silicon, or, a imho more apples-to-apples comparison, e.g., amd strixpoint halo vs apple silicon?

i would expect the "pure" cpu inference to be behind, but an approach like strix halo/dgx spark to be much closer?

Re: Ollama is now powered by MLX on Apple Silicon in preview

#74

Post by miki123211 »

babblingfish wrote: post #2

LLMs on device is the future. It's more secure and solves the problem of too much demand for inference compared to data center supply, it also would use less electricity. It's just a matter of getting the performance good enough. Most users don't need frontier model performance.

> would use less electricity

Sorry to shatter your bubble, but this is patently false, LLMs are far more efficient on hardware that simultaneously serves many requests at once.

There's also the (environmental and monetary) cost of producing overpowered devices that sit idle when you're not using them, in contrast to a cloud GPU, which can be rented out to whoever needs it at a given moment, potentially at a lower cost during periods of lower demand.

Many LLM workloads aren't even that latency sensitive, so it's far easier to move them closer to renewable energy than to move that energy closer to you.

Re: Ollama is now powered by MLX on Apple Silicon in preview

#75

Post by thih9 »

babblingfish wrote: post #2

LLMs on device is the future. It's more secure and solves the problem of too much demand for inference compared to data center supply, it also would use less electricity. It's just a matter of getting the performance good enough. Most users don't need frontier model performance.

> it also would use less electricity

How would it use less electricity? I’d like to learn more.

Re: Ollama is now powered by MLX on Apple Silicon in preview

#76

Post by kortilla »

miki123211 wrote: post #74
babblingfish wrote: post #2

LLMs on device is the future. It's more secure and solves the problem of too much demand for inference compared to data center supply, it also would use less electricity. It's just a matter of getting the performance good enough. Most users don't need frontier model performance.

> would use less electricity Sorry to shatter your bubble, but this is patently false, LLMs are far more efficient on hardware that simultaneously serves many requests at once. There's also the (environmental and monetary) cost of producing overpowered devices that sit idle when you're not using them, in contrast to a cloud GPU, which can be rented out to whoever needs it at a given moment, potentially at a lower cos…

Well this is an article about running on hardware I already have in my house. In the winter that’s just a little extra electricity that converts into “free” resistive heating.

Re: Ollama is now powered by MLX on Apple Silicon in preview

#77

Post by jychang »

thih9 wrote: post #75
babblingfish wrote: post #2

LLMs on device is the future. It's more secure and solves the problem of too much demand for inference compared to data center supply, it also would use less electricity. It's just a matter of getting the performance good enough. Most users don't need frontier model performance.

> it also would use less electricity How would it use less electricity? I’d like to learn more.

That's completely not true. LLM on device would use MORE electricity.

Service providers that do batch>1 inference are a lot more efficient per watt.

Local inference can only do batch=1 inference, which is very inefficient.

Re: Ollama is now powered by MLX on Apple Silicon in preview

#78

Post by kortilla »

raincole wrote: post #37
aurareturn wrote: post #9

Earlier quoted context omitted.

It isn't going to replace cloud LLMs since cloud LLMs will always be faster in throughput and smarter. Cloud and local LLMs will grow together, not replace each other. I'm not convinced that local LLMs use less electricity either. Per token at the same level of intelligence, cloud LLMs should run circles around local LLMs in efficiency. If it doesn't, what are we paying hundreds of billions of dollars for? I think lo…

Yep. People were claiming DeepSeek was "almost as good as SOTA" when it came out. Local will always be one step away like fusion. It's just wishful thinking (and hatred towards American megacorps). Old as the hills. Understandable, but not based on reality.

Don’t try to draw trend lines for an industry that has existed for <5 years.

Re: Ollama is now powered by MLX on Apple Silicon in preview

#79

Post by jychang »

Barbing wrote: post #64
asutekku wrote: post #48

Earlier quoted context omitted.

Frontier model has much better knowledge and they usually hallucinate less. It's not about the coding capabilities, it's about how much you can trust the model.

re: trust- Have you tried the free version of ChatGPT? It is positively appalling. It’s like GPT 3.5 but prompted to write three times as much as necessary to seem useful. I wonder how many people have embarrassed themselves, lost their jobs, and been critically misinformed. All easy with state-of-the-art models but seemingly a guarantee with the bottom sub-slop tier. Is the average person just talking to it about th…

The free version of ChatGPT is insanely crippled, so that's not surprising.

Re: Ollama is now powered by MLX on Apple Silicon in preview

#80

Post by tomashubelbauer »

ZeroGravitas wrote: post #67
babblingfish wrote: post #2

LLMs on device is the future. It's more secure and solves the problem of too much demand for inference compared to data center supply, it also would use less electricity. It's just a matter of getting the performance good enough. Most users don't need frontier model performance.

It feels like you'll soon need a local llm to intermediate with the remote llm, like an ad blocker for browsers to stop them injecting ads or remind you not to send corporate IP out onto the Internet.

I'd like to coin the term "user agent" for this
Post reply on HN