Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Ollama is now powered by MLX on Apple Silicon in preview

ollama.com

Post reply on HN
Print view View original

251–260 of 384 posts

Re: Ollama is now powered by MLX on Apple Silicon in preview

#251

Post by raw_anon_1111 »

abu_ameena wrote: post #247
raw_anon_1111 wrote: post #234

Earlier quoted context omitted.

Users don’t care about “privacy”. If they did, Meta and Alphabet wouldn’t be worth $1T+. Users really don’t matter at all. The revenue for AI companies will be B2B where the user is not the customer - including coding agents. Most people don’t even use computers as their primary “computing device” and most people are buying crappy low end Android phones - no I’m not saying all Android phones are crappy. But that’s wh…

I see it as a long-term tradeoff on user freedom. You pay upfront for a capable hardware, you get your services running locally (you don’t pay subscriptions). Or you buy cheap hardware, you still need the same services “running in some cloud” for $X monthly. X goes up depending on the corporate bottom-line

In the history of cloud computing, prices have mostly only come down especially as inference becomes a commodity. Realistically, just looking at Mac prices, the cost of a computer with decent local inference would be around $6000 per person.

The world is not moving back to on prem.

Re: Ollama is now powered by MLX on Apple Silicon in preview

#252

Post by bwfan123 »

What is the cheapest usable local rig for coding ? I dont want fancy agents and such, but something purpose built for coders, and fast-enough for my use, and open-source, so I can tweak it to my liking. Things are moving fast, and I am hesitant to put in 3-4K now in the hope that it would be cheaper if i wait.

Re: Ollama is now powered by MLX on Apple Silicon in preview

#253

Post by Kichererbsen »

drob518 wrote: post #176
zozbot234 wrote: post #172

Earlier quoted context omitted.

> it can take anywhere between 6-25 seconds for a response (after lots of thinking) from me asking "Hello world". Qwen thinking likes to second-guess itself a LOT when faced with simple/vague prompts like that. (I'll answer it this way. Generating output. Wait, I'll answer it that way. Generating output. Wait, I'll answer it this way... lather, rinse, repeat.) I suppose this is their version of "super smart fancy thi…

Indeed. Qwen doesn’t just second guess itself, it third and fourth guesses itself.

Solid Terry Pratchett reference right there.

Re: Ollama is now powered by MLX on Apple Silicon in preview

#254

Post by theChaparral »

jonhohle wrote: post #220
babblingfish wrote: post #2

LLMs on device is the future. It's more secure and solves the problem of too much demand for inference compared to data center supply, it also would use less electricity. It's just a matter of getting the performance good enough. Most users don't need frontier model performance.

I’ve been using google search AI and Gemini, which I find generally pretty good. In the past week, Gemini and Search AI have been bringing in various details of previous searches I’ve done and Search AI conversations I’ve had and it’s extremely gross and creepy. I was looking for details about cars and it started interjecting how the safety would affect my children by name in a conversation where I never mention my c…

It's Personal Intelligence in the Gemini settings. I just turned that off last night when it was doing similar things.

Re: Ollama is now powered by MLX on Apple Silicon in preview

#255

Post by 0xc133 »

braum wrote: post #236

How does Ollama help with Claude Code? Claude code runs in terminal but AFAIK connects back to anthropic directly and cannot run locally. I hope I'm missing something obvious.

https://docs.ollama.com/integrations/claude-code

You can use models like qwen3.5 running on local hardware in ollama and redirect Claude to use the local ollama API endpoint instead of Anthropic’s servers.

Re: Ollama is now powered by MLX on Apple Silicon in preview

#256

Post by esseph »

raw_anon_1111 wrote: post #251
abu_ameena wrote: post #247

Earlier quoted context omitted.

I see it as a long-term tradeoff on user freedom. You pay upfront for a capable hardware, you get your services running locally (you don’t pay subscriptions). Or you buy cheap hardware, you still need the same services “running in some cloud” for $X monthly. X goes up depending on the corporate bottom-line

In the history of cloud computing, prices have mostly only come down especially as inference becomes a commodity. Realistically, just looking at Mac prices, the cost of a computer with decent local inference would be around $6000 per person. The world is not moving back to on prem.

> The world is not moving back to on prem.

Lol, you should tell my customers (that are moving back on prem) that!

You should also tell Microsoft, who just yesterday said they are going back to focusing on local apps.

Re: Ollama is now powered by MLX on Apple Silicon in preview

#257

Post by dec0dedab0de »

testing22321 wrote: post #238
abu_ameena wrote: post #223

On-device models are the future. Users prefer them. No privacy issues. No dealing with connectivity, tokens, or changes to vendors implementations. I have an app using Foundation Model, and it works great. I only wish I could backport it to pre macOS 26 versions.

I see all these LLM posts about if a certain model can run locally on certain hardware and I don’t get it. What are you doing with these local models that run at x tokens/sec. Do you have the equivalent of ChatGPT running entirely locally? What do you do with it? Why? I honestly don’t understand the point or use case.

most of the llm tooling can handle different models. Ollama makes it easy to install and run different models locally. So you can configure aider or vscode or whatever you're using to connect to chatgpt to point to your local models instead.

None of them are as good as the big hosted models, but you might be surprised at how capable they are. I like running things locally when I can, and I also like not worrying about accidentally burning through tokens.

I think the future is multiple locally run models that call out to hosted models when necessary. I can imagine every device coming with a base model and using loras to learn about the users needs. With companies and maybe even households having their own shared models that do heavier lifting. while companies like openai and anhtropic continue to host the most powerful and expensive options.

Re: Ollama is now powered by MLX on Apple Silicon in preview

#258

Post by xiphias2 »

bwfan123 wrote: post #252

What is the cheapest usable local rig for coding ? I dont want fancy agents and such, but something purpose built for coders, and fast-enough for my use, and open-source, so I can tweak it to my liking. Things are moving fast, and I am hesitant to put in 3-4K now in the hope that it would be cheaper if i wait.

It doesn't look like RAM, CPU GPU or bandwidth is getting cheaper if that helps you, quite the opposite.

Re: Ollama is now powered by MLX on Apple Silicon in preview

#259

Post by fredoliveira »

mirekrusin wrote: post #55
aurareturn wrote: post #9

Earlier quoted context omitted.

It isn't going to replace cloud LLMs since cloud LLMs will always be faster in throughput and smarter. Cloud and local LLMs will grow together, not replace each other. I'm not convinced that local LLMs use less electricity either. Per token at the same level of intelligence, cloud LLMs should run circles around local LLMs in efficiency. If it doesn't, what are we paying hundreds of billions of dollars for? I think lo…

Local RTX 5090 is actually faster than A100/H100.

Crazy thing to say without other contextual information - it obviously depends on a number of factors. Do you have an apples to apples comparison at hand?

Re: Ollama is now powered by MLX on Apple Silicon in preview

#260

Post by DesiLurker »

raw_anon_1111 wrote: post #234
abu_ameena wrote: post #223

On-device models are the future. Users prefer them. No privacy issues. No dealing with connectivity, tokens, or changes to vendors implementations. I have an app using Foundation Model, and it works great. I only wish I could backport it to pre macOS 26 versions.

Users don’t care about “privacy”. If they did, Meta and Alphabet wouldn’t be worth $1T+. Users really don’t matter at all. The revenue for AI companies will be B2B where the user is not the customer - including coding agents. Most people don’t even use computers as their primary “computing device” and most people are buying crappy low end Android phones - no I’m not saying all Android phones are crappy. But that’s wh…

you are missing a but 'given a choice' disclaimer. Meta is pretty much a monopoly in social space. So is Android. given a choice people will absolutely gravitate towards not-always-snooping device. most people with resources anyway, who matter for the AI adoption.

Oh an wait till ad companies start selling your healthcare data and you will see how fast things turn 'given a choice'.

Post reply on HN