Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Ollama is now powered by MLX on Apple Silicon in preview

ollama.com

Post reply on HN
Print view View original

301–310 of 384 posts

Re: Ollama is now powered by MLX on Apple Silicon in preview

#301

Post by ilovecake1984 »

raw_anon_1111 wrote: post #234
abu_ameena wrote: post #223

On-device models are the future. Users prefer them. No privacy issues. No dealing with connectivity, tokens, or changes to vendors implementations. I have an app using Foundation Model, and it works great. I only wish I could backport it to pre macOS 26 versions.

Users don’t care about “privacy”. If they did, Meta and Alphabet wouldn’t be worth $1T+. Users really don’t matter at all. The revenue for AI companies will be B2B where the user is not the customer - including coding agents. Most people don’t even use computers as their primary “computing device” and most people are buying crappy low end Android phones - no I’m not saying all Android phones are crappy. But that’s wh…

Users here probably means corporations. I still don’t see much use of LLMs in my personal life, other than one thing. Googling stuff in a foreign language.

Re: Ollama is now powered by MLX on Apple Silicon in preview

#302

Post by api »

raw_anon_1111 wrote: post #234
abu_ameena wrote: post #223

On-device models are the future. Users prefer them. No privacy issues. No dealing with connectivity, tokens, or changes to vendors implementations. I have an app using Foundation Model, and it works great. I only wish I could backport it to pre macOS 26 versions.

Users don’t care about “privacy”. If they did, Meta and Alphabet wouldn’t be worth $1T+. Users really don’t matter at all. The revenue for AI companies will be B2B where the user is not the customer - including coding agents. Most people don’t even use computers as their primary “computing device” and most people are buying crappy low end Android phones - no I’m not saying all Android phones are crappy. But that’s wh…

"Users" is a large set of people. Many don't care about privacy, but some do. There's also a difference between where you post random social media stuff vs what you run with something like OpenClaw and give access to your machine.

Re: Ollama is now powered by MLX on Apple Silicon in preview

#303

Post by abu_ameena »

mgaunard wrote: post #292
abu_ameena wrote: post #223

On-device models are the future. Users prefer them. No privacy issues. No dealing with connectivity, tokens, or changes to vendors implementations. I have an app using Foundation Model, and it works great. I only wish I could backport it to pre macOS 26 versions.

These local models are far behind the capabilities of latest Gemini Pro, Claude Opus or GPT. Why waste time with subpar AI?

Yes, but you don’t always want the power/expense of these models for the task at hand. A hammer is good enough to push a nail inside a wall. Save the nail gun for when you are building a house.

Re: Ollama is now powered by MLX on Apple Silicon in preview

#304

Post by nozzlegear »

raw_anon_1111 wrote: post #261
DesiLurker wrote: post #260

Earlier quoted context omitted.

you are missing a but 'given a choice' disclaimer. Meta is pretty much a monopoly in social space. So is Android. given a choice people will absolutely gravitate towards not-always-snooping device. most people with resources anyway, who matter for the AI adoption. Oh an wait till ad companies start selling your healthcare data and you will see how fast things turn 'given a choice'.

People A) don’t have to use Meta and B) do have a choice between not using a mobile phone by an ad tech company.

People don't have a choice between Facebook and not-Facebook-but-still-has-all-of-your-friends-and-family. Abstinence isn't a choice here any more than shutting off your cell phone service is a choice; true in the literal sense, but only if you don't mind being unreachable to everyone who still has a phone.

Re: Ollama is now powered by MLX on Apple Silicon in preview

#305

Post by yg1112 »

LuxBennu wrote: post #7

Already running qwen 70b 4-bit on m2 max 96gb through llama.cpp and it's pretty solid for day to day stuff. The mlx switch is interesting because ollama was basically shelling out to llama.cpp on mac before, so native mlx should mean better memory handling on apple silicon. Curious to see how it compares on the bigger models vs the gguf path

The key difference is that MLX's array model assumes unified memory from the ground up. llama.cpp's Metal backend works fine but carries abstractions from the discrete GPU world — explicit buffer synchronization, command buffer boundaries — that are unnecessary when CPU and GPU share the same address space. You'll notice the gap most at large context lengths where KV cache pressure is highest.

Re: Ollama is now powered by MLX on Apple Silicon in preview

#306

Post by charcircuit »

roadside_picnic wrote: post #285
raw_anon_1111 wrote: post #234

Earlier quoted context omitted.

Users don’t care about “privacy”. If they did, Meta and Alphabet wouldn’t be worth $1T+. Users really don’t matter at all. The revenue for AI companies will be B2B where the user is not the customer - including coding agents. Most people don’t even use computers as their primary “computing device” and most people are buying crappy low end Android phones - no I’m not saying all Android phones are crappy. But that’s wh…

> Users don’t care about “privacy”. I worked for a research focused AI startup that had a strict "no external LLM" policy for code touching our core research. You're right that the average consumer doesn't care about privacy, but there are many, many users who do. The average consumer also don't have a desktop with GPU or high end Mac Studio, but that doesn't mean there aren't many people working with AI how do have…

Those users are addressed by being able to rent their own exclusive machines to run the model on. There will be some compromise that will be made to get access to the best intelligence available.

Re: Ollama is now powered by MLX on Apple Silicon in preview

#307

Post by Nevermark »

raw_anon_1111 wrote: post #234
abu_ameena wrote: post #223

On-device models are the future. Users prefer them. No privacy issues. No dealing with connectivity, tokens, or changes to vendors implementations. I have an app using Foundation Model, and it works great. I only wish I could backport it to pre macOS 26 versions.

Users don’t care about “privacy”. If they did, Meta and Alphabet wouldn’t be worth $1T+. Users really don’t matter at all. The revenue for AI companies will be B2B where the user is not the customer - including coding agents. Most people don’t even use computers as their primary “computing device” and most people are buying crappy low end Android phones - no I’m not saying all Android phones are crappy. But that’s wh…

Have you done A/B tests to see if consumers prefer Facebook with or without privacy?

No? What? Oh, you can't?

Neither can consumers. Most consumers are very aware of the lack of privacy, the manipulation, and have very cynical feelings about Facebook and similar companies. But it's where their friends and family are.

For most people the web is a mine field maze where basic things they want are compromised everywhere. And they are routinely creeped out by ads that reveal they know them far too personally.

You are mistaking network capture for preference.

Another telling example. Lots of privacy valuing technical people, who would never have a Facebook account, send unencrypted text emails.

It is network capture, not preference.

Re: Ollama is now powered by MLX on Apple Silicon in preview

#308

Post by barkerja »

raw_anon_1111 wrote: post #234
abu_ameena wrote: post #223

On-device models are the future. Users prefer them. No privacy issues. No dealing with connectivity, tokens, or changes to vendors implementations. I have an app using Foundation Model, and it works great. I only wish I could backport it to pre macOS 26 versions.

Users don’t care about “privacy”. If they did, Meta and Alphabet wouldn’t be worth $1T+. Users really don’t matter at all. The revenue for AI companies will be B2B where the user is not the customer - including coding agents. Most people don’t even use computers as their primary “computing device” and most people are buying crappy low end Android phones - no I’m not saying all Android phones are crappy. But that’s wh…

User's care about privacy when they understand the threat and impact. The issue is most user's don't understand this, especially when it comes to use of products like Meta where on the surface, everything appears harmless.

Re: Ollama is now powered by MLX on Apple Silicon in preview

#309

Post by newman314 »

franze wrote: post #86

I created "apfel" https://github.com/Arthur-Ficial/apfel a CLI for the apple on-device local foundation model (Apple intelligence) yeah its super limited with its 4k context window and super common false positives guardrails (just ask it to describe a color) ... bit still ... using it in bash scripts that just work without calling home / out or incurring extra costs feels super powerful.

This is quite interesting. I wonder if AFM is smart enough to do spam classification.

Re: Ollama is now powered by MLX on Apple Silicon in preview

#310

Post by subarctic »

peronperon wrote: post #283
DevKoan wrote: post #282

The Foundation Model point is real. As an iOS developer, what excites me most isn't the performance — it's what on-device inference does to the app architecture. When you're not making network calls, you stop thinking in "loading states" and start thinking in "local state machines." The UX design space opens up completely. Interactions that felt too fast to justify a server round-trip are suddenly viable. The backpor…

Don't post generated comments or AI-edited comments. HN is for conversation between humans. https://news.ycombinator.com/newsguidelines.html#comments

What gave this one away — just the em dashes?
Post reply on HN