Ollama is now powered by MLX on Apple Silicon in preview

Post by raw_anon_1111 » Tue, Mar 31, 2026, 7:35 PM UTC

Nevermark wrote: post #327

raw_anon_1111 wrote: post #319

Earlier quoted context omitted.

Consumers pro actively tell Facebook their age, sexual preference, race, relationship status, likes and dislikes, they check in to where they are and who they are there with… They are choosing to give Facebook info.

> They are choosing to give Facebook info. Yes, they do. That's is exactly the phenomena my comment addressed. But the way you wrote that implies an improbable motivation or choice framing. Perhaps their real motive/choice is to share with other people on the site. It is called a network effect. If (1) Facebook had been the surveillance/manipulation capital of the world from inception, (2) an equally inviting privacy…

Post by raw_anon_1111 » Tue, Mar 31, 2026, 8:09 PM UTC

Nevermark wrote: post #332

raw_anon_1111 wrote: post #331

Earlier quoted context omitted.

Wouldn’t the most obvious way for people to protect their privacy while using FB if they cared and still wanted to use FB be not to proactively give them information? You don’t have to share everything I mentioned just to be involved in a group.

> You don’t have to share everything I mentioned just to be involved in a group. This is clearly true. There is an implied point here but I am not sure what. They share in their profile what they want other people to see. And often choose to not fill out everything. Nobody signs up to share with Meta, Inc. Most people would love a "[ ] Do not share with Facebook". People choosing an imperfect option, from imperfect o…

Post by lioeters » Tue, Mar 31, 2026, 8:38 PM UTC

yg1112 wrote: post #305

LuxBennu wrote: post #7

Already running qwen 70b 4-bit on m2 max 96gb through llama.cpp and it's pretty solid for day to day stuff. The mlx switch is interesting because ollama was basically shelling out to llama.cpp on mac before, so native mlx should mean better memory handling on apple silicon. Curious to see how it compares on the bigger models vs the gguf path

The key difference is that MLX's array model assumes unified memory from the ground up. llama.cpp's Metal backend works fine but carries abstractions from the discrete GPU world — explicit buffer synchronization, command buffer boundaries — that are unnecessary when CPU and GPU share the same address space. You'll notice the gap most at large context lengths where KV cache pressure is highest.

Post by duxup » Tue, Mar 31, 2026, 8:46 PM UTC

raw_anon_1111 wrote: post #234

abu_ameena wrote: post #223

On-device models are the future. Users prefer them. No privacy issues. No dealing with connectivity, tokens, or changes to vendors implementations. I have an app using Foundation Model, and it works great. I only wish I could backport it to pre macOS 26 versions.

Users don’t care about “privacy”. If they did, Meta and Alphabet wouldn’t be worth $1T+. Users really don’t matter at all. The revenue for AI companies will be B2B where the user is not the customer - including coding agents. Most people don’t even use computers as their primary “computing device” and most people are buying crappy low end Android phones - no I’m not saying all Android phones are crappy. But that’s wh…

Post by testing22321 » Tue, Mar 31, 2026, 8:48 PM UTC

samuel wrote: post #243

testing22321 wrote: post #238

Earlier quoted context omitted.

I see all these LLM posts about if a certain model can run locally on certain hardware and I don’t get it. What are you doing with these local models that run at x tokens/sec. Do you have the equivalent of ChatGPT running entirely locally? What do you do with it? Why? I honestly don’t understand the point or use case.

Chat is certainly an option, but the real deal are agents, which have access to way more sensitive information.