Ollama is now powered by MLX on Apple Silicon in preview • Hacker News Board

Re: Ollama is now powered by MLX on Apple Silicon in preview

#211

Post by susupro1 » Tue, Mar 31, 2026, 1:24 PM UTC

gedy wrote: post #5

babblingfish wrote: post #2

LLMs on device is the future. It's more secure and solves the problem of too much demand for inference compared to data center supply, it also would use less electricity. It's just a matter of getting the performance good enough. Most users don't need frontier model performance.

Man I really hope so, as, as much as I like Claude Code, I hate the company paying for it and tracking your usage, bullshit management control, etc. I feel like I'm training my replacement. Things feel like they are tightening vs more power and freedom. On device I would gladly pay for good hardware - it's my machine and I'm using as I see fit like an IDE.

[dead]

Re: Ollama is now powered by MLX on Apple Silicon in preview

#212

Post by jedahan » Tue, Mar 31, 2026, 1:27 PM UTC

franze wrote: post #192

woadwarrior01 wrote: post #139

Earlier quoted context omitted.

There's a very similar afm CLI that can be installed via Homebrew. https://github.com/scouzi1966/maclocal-api

done brew tap Arthur-Ficial/tap brew install Arthur-Ficial/tap/apfel

No need for the extra tap step, this works fine alone:

    brew install Arthur-Ficial/tap/apfel

Re: Ollama is now powered by MLX on Apple Silicon in preview

#213

Post by jorvi » Tue, Mar 31, 2026, 1:31 PM UTC

franze wrote: post #125

LeoDaVibeci wrote: post #116

Earlier quoted context omitted.

Honestly I can't believe Apple put that foundation model product out the door. I was so excited about it, but when I tried it, it was such a disappointment. Glad to hear you calling that out so I know it wasn't just me. Looks like they have pivoted completely over to Gemini, thank god.

yeah, it is super limited but also you can now do cmd(){ local x c r a; while [[ $1 == -* ]]; do case $1 in -x)x=1;shift;; -c)c=1;shift;; *)break;; esac; done; r=$(apfel -q -s 'Output only a shell command.' "$*" | sed '/^```/d;/^#/d;s/^[[:space:]]*//;/^$/d' | head -1); [[ $r ]] || { echo "no command generated"; return 1; }; printf '\e[32m$\e[0m %s\n' "$r"; [[ $c ]] && printf %s "$r" | pbcopy && echo "(copied)"; [[ $x…

This really makes me think of A Deepness in the Sky by Vernor Vinge. A loose prequel to A Fire Upon The Deep, and IMO actually the superior story. It plays in the far future of humanity.

In part of it, one group tries to take control of a huge ship from another group. They in part do this by trying to bypass all the cybersecurity. But in those far future days, you don't interface with all the aeons of layers of command protocols anymore, you just query an AI who does it for you. So, this group has a few tech guys that try the bypass by using the old command protocols directly (in a way the same thing like the iOS exploit that used a vulnerability in a PostScript font library from 90s).

Imagine being used to LLM prompting + responses, and suddenly you have to deal with something like

  sed '/^```/d;/^#/d;s/^[[:space:]]\*//;/^$/d' | head -1); [[ $r ]]

and generally obtuse terminal output and man pages.

:)

(offtopic: name your variables, don't do local x c r a;. Readability is king, and a few hundred thousand years from now some poor Qeng Ho fellow might thank his lucky stars you did).

Re: Ollama is now powered by MLX on Apple Silicon in preview

#214

Post by susupro1 » Tue, Mar 31, 2026, 1:31 PM UTC

theshrike79 wrote: post #128

girvo wrote: post #121

Earlier quoted context omitted.

I'd recommend it too, because the knowledge cutoff of all the open weight Chinese models (M2.7, Qwen3.5, GLM-5 etc) is earlier than you'd think, so giving it web search (I use `ddgr` with a skill) helps a surprising amount

Yep, having a "stupid" central model with multiple tools is IMO the key to efficient agentic systems. It needs to be just smart enough to use the tools and distill the responses into something usable. And one of the tools can be "ask claude/codex/gemini" so the local model itself doesn't actually need to do much.

[dead]

Re: Ollama is now powered by MLX on Apple Silicon in preview

#215

Post by chid » Tue, Mar 31, 2026, 1:33 PM UTC

franze wrote: post #86

I created "apfel" https://github.com/Arthur-Ficial/apfel a CLI for the apple on-device local foundation model (Apple intelligence) yeah its super limited with its 4k context window and super common false positives guardrails (just ask it to describe a color) ... bit still ... using it in bash scripts that just work without calling home / out or incurring extra costs feels super powerful.

this is real neat. I'll give it a spin.

Re: Ollama is now powered by MLX on Apple Silicon in preview

#216

Post by corndoge » Tue, Mar 31, 2026, 1:33 PM UTC

dgacmu wrote: post #210

beepbooptheory wrote: post #202

Earlier quoted context omitted.

What is the AI doing here? Or is this just like being cheeky?

The pile of shell and sed is cleaning up the ai output and then running it in the shell. The instruction to the AI was to create _a_ shell command. So it's a random shell command generator (maybe).

that part is the system prompt, the script is a function that takes a prompt describing a shell command as an argument

Re: Ollama is now powered by MLX on Apple Silicon in preview

#217

Post by fooker » Tue, Mar 31, 2026, 1:42 PM UTC

domh wrote: post #165

I have an M4 Max with 48GB RAM. Anyone have any tips for good local models? Context length? Using the model recommended in the blog post (qwen3.5:35b-a3b-coding-nvfp4) with Ollama 0.19.0 and it can take anywhere between 6-25 seconds for a response (after lots of thinking) from me asking "Hello world". Is this the best that's currently achievable with my hardware or is there something that can be configured to get bet…

Avoid reasoning models in any situation where you have low tokens/second

Re: Ollama is now powered by MLX on Apple Silicon in preview

#218

Post by a-dub » Tue, Mar 31, 2026, 1:43 PM UTC

is local llm inference on modern macbook pros comfortable yet? when i played with it a year or so ago, it worked fairly ok but definitely produced uncomfortable levels of heat.

(regarding mlx, there were toolkits built on mlx that supported qlora fine tuning and inference, but also produced a bunch of heat)

Re: Ollama is now powered by MLX on Apple Silicon in preview

#219

Post by lxgr » Tue, Mar 31, 2026, 1:45 PM UTC

throwaway27448 wrote: post #102

lxgr wrote: post #70

Earlier quoted context omitted.

Have you used GPT instant or mini yourself? I think it’s pretty cynical to assume that this is “good enough for most people”, even if they don’t know the difference between that and better models.

Say more. Why do you think this?

They're essentially replying based on vibes, instead of grounding their responses in extensive web searches, which is what the paid models/configurations generally do. This makes them wrong more often than they're right for anything but the most trivial requests that can be easily responded to out of memorized training data.

This is all on top of the (to me) insufferable tone of the non-thinking models, but that might well be how most users prefer to be talked to, and whether that's how these models should accordingly talk is a much more nuanced question.

Regardless of that, everybody deserves correct answers, even users on the free tier. If this makes the free tier uneconomical to serve for hours on end per user per day, then I'd much rather they limit the number of turns than dial down the quality like that.

Re: Ollama is now powered by MLX on Apple Silicon in preview

#220

Post by jonhohle » Tue, Mar 31, 2026, 1:54 PM UTC

babblingfish wrote: post #2

LLMs on device is the future. It's more secure and solves the problem of too much demand for inference compared to data center supply, it also would use less electricity. It's just a matter of getting the performance good enough. Most users don't need frontier model performance.

I’ve been using google search AI and Gemini, which I find generally pretty good. In the past week, Gemini and Search AI have been bringing in various details of previous searches I’ve done and Search AI conversations I’ve had and it’s extremely gross and creepy.

I was looking for details about cars and it started interjecting how the safety would affect my children by name in a conversation where I never mention my children. I was asking details about Thunderbolt and modern Ryzen processors and a fresh Gemini chat brought in details about a completely unrelated project I work on. I’ve always thought local LLMs would be important, but whatever Google did in the past few weeks has made that even more clear.