Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Ollama is now powered by MLX on Apple Silicon in preview

ollama.com

Post reply on HN
Print view View original

241–250 of 384 posts

Re: Ollama is now powered by MLX on Apple Silicon in preview

#241

Post by barelysapient »

raw_anon_1111 wrote: post #234
abu_ameena wrote: post #223

On-device models are the future. Users prefer them. No privacy issues. No dealing with connectivity, tokens, or changes to vendors implementations. I have an app using Foundation Model, and it works great. I only wish I could backport it to pre macOS 26 versions.

Users don’t care about “privacy”. If they did, Meta and Alphabet wouldn’t be worth $1T+. Users really don’t matter at all. The revenue for AI companies will be B2B where the user is not the customer - including coding agents. Most people don’t even use computers as their primary “computing device” and most people are buying crappy low end Android phones - no I’m not saying all Android phones are crappy. But that’s wh…

Different users. Many people care about privacy and aren’t using Meta products. And many businesses care about it too and have information policies to protect their IP.

Re: Ollama is now powered by MLX on Apple Silicon in preview

#242

Post by navigate8310 »

braum wrote: post #236

How does Ollama help with Claude Code? Claude code runs in terminal but AFAIK connects back to anthropic directly and cannot run locally. I hope I'm missing something obvious.

I believe one can use the CC as the primary model driving local agents that use local models

Re: Ollama is now powered by MLX on Apple Silicon in preview

#243

Post by samuel »

testing22321 wrote: post #238
abu_ameena wrote: post #223

On-device models are the future. Users prefer them. No privacy issues. No dealing with connectivity, tokens, or changes to vendors implementations. I have an app using Foundation Model, and it works great. I only wish I could backport it to pre macOS 26 versions.

I see all these LLM posts about if a certain model can run locally on certain hardware and I don’t get it. What are you doing with these local models that run at x tokens/sec. Do you have the equivalent of ChatGPT running entirely locally? What do you do with it? Why? I honestly don’t understand the point or use case.

Chat is certainly an option, but the real deal are agents, which have access to way more sensitive information.

Re: Ollama is now powered by MLX on Apple Silicon in preview

#244

Post by xmddmx »

On a M4 Pro MacBook Pro with 48GB RAM I did this test:

ollama run $model "calculate fibonacci numbers in a one-line bash script" --verbose

  Model                         PromptEvalRate EvalRate
  ------------------------------------------------------
  qwen3.5:35b-a3b-q4_K_M         6.6            30.0
  qwen3.5:35b-a3b-nvfp4         13.2            66.5
  qwen3.5:35b-a3b-int4          59.4            84.4
I can't comment on the quality differences (if any) between these three.

Re: Ollama is now powered by MLX on Apple Silicon in preview

#245

Post by samuel »

braum wrote: post #236

How does Ollama help with Claude Code? Claude code runs in terminal but AFAIK connects back to anthropic directly and cannot run locally. I hope I'm missing something obvious.

You can connect it to any anthropic compatible endpoint(kimi allows this) but it's a weird choice, given that Open code, pi.dev and others are open source.

Re: Ollama is now powered by MLX on Apple Silicon in preview

#246

Post by raw_anon_1111 »

barelysapient wrote: post #241
raw_anon_1111 wrote: post #234

Earlier quoted context omitted.

Users don’t care about “privacy”. If they did, Meta and Alphabet wouldn’t be worth $1T+. Users really don’t matter at all. The revenue for AI companies will be B2B where the user is not the customer - including coding agents. Most people don’t even use computers as their primary “computing device” and most people are buying crappy low end Android phones - no I’m not saying all Android phones are crappy. But that’s wh…

Different users. Many people care about privacy and aren’t using Meta products. And many businesses care about it too and have information policies to protect their IP.

70% of the world’s population use at least one Meta property at least once per day. How many of the other 30% are too poor/young/computer illiterate to be part of an addressable market?

Every company has dozens of SaaS products that store their business critical information. Amazon installs Office on each computer, Slack (they were moving away from Chime when I left), and the sales department uses SalesForce - SA’s and Professional Services (former employee).

The addressable market of even companies that care about privacy is not a large addressable market. How long will it be before computers become cheap enough that can run even GPT 4 level LLMs that companies will give it to all of their developers?

Re: Ollama is now powered by MLX on Apple Silicon in preview

#247

Post by abu_ameena »

raw_anon_1111 wrote: post #234
abu_ameena wrote: post #223

On-device models are the future. Users prefer them. No privacy issues. No dealing with connectivity, tokens, or changes to vendors implementations. I have an app using Foundation Model, and it works great. I only wish I could backport it to pre macOS 26 versions.

Users don’t care about “privacy”. If they did, Meta and Alphabet wouldn’t be worth $1T+. Users really don’t matter at all. The revenue for AI companies will be B2B where the user is not the customer - including coding agents. Most people don’t even use computers as their primary “computing device” and most people are buying crappy low end Android phones - no I’m not saying all Android phones are crappy. But that’s wh…

I see it as a long-term tradeoff on user freedom. You pay upfront for a capable hardware, you get your services running locally (you don’t pay subscriptions). Or you buy cheap hardware, you still need the same services “running in some cloud” for $X monthly. X goes up depending on the corporate bottom-line

Re: Ollama is now powered by MLX on Apple Silicon in preview

#249

Post by austinthetaco »

troad wrote: post #51
babblingfish wrote: post #2

LLMs on device is the future. It's more secure and solves the problem of too much demand for inference compared to data center supply, it also would use less electricity. It's just a matter of getting the performance good enough. Most users don't need frontier model performance.

I very recently installed llama.cpp on my consumer-grade M4 MBP, and I've been having loads of fun poking and prodding the local models. There's now a ChatGPT style interface baked into llama.cpp, which is very handy for quick experimentation. (I'm not entirely sure what Ollama would get me that llama.cpp doesn't, happy to hear suggestions!) There are some surprisingly decent models that happily fit even into a mere…

Have you played around with any of the Hermes models? they are supposed to be one of the best at non-refusal while keeping sane.

Re: Ollama is now powered by MLX on Apple Silicon in preview

#250

Post by adam_patarino »

podlp wrote: post #232
adam_patarino wrote: post #204

Earlier quoted context omitted.

[flagged]

Rig sounds cool, I just joined the waitlist! I’m building something similar although with a much narrower purpose. Excited to learn more

Tell me more! Thanks for the waitlist
Post reply on HN