Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Ollama is now powered by MLX on Apple Silicon in preview

ollama.com

Post reply on HN
Print view View original

181–190 of 384 posts

Re: Ollama is now powered by MLX on Apple Silicon in preview

#181

Post by dgb23 »

aurareturn wrote: post #12
gedy wrote: post #5

Earlier quoted context omitted.

Man I really hope so, as, as much as I like Claude Code, I hate the company paying for it and tracking your usage, bullshit management control, etc. I feel like I'm training my replacement. Things feel like they are tightening vs more power and freedom. On device I would gladly pay for good hardware - it's my machine and I'm using as I see fit like an IDE.

When local LLMs get good enough for you to use delightfully, cloud LLMs will have gotten so much smarter that you'll still use it for stuff that needs more intelligence.

That's not necessarily the case. So far, commercial cloud LLMs have maintained a head-start, but there is no law of nature that prevents us from having competitive open models.

In fact the space seems to move at a rapid pace as more and more specialized models come out. There's a possible trajectory where open weight models will compete side by side or even be preferable for many use cases, just like what happened with OS's and SQL DB's.

Re: Ollama is now powered by MLX on Apple Silicon in preview

#182

Post by domh »

xienze wrote: post #177
domh wrote: post #165

I have an M4 Max with 48GB RAM. Anyone have any tips for good local models? Context length? Using the model recommended in the blog post (qwen3.5:35b-a3b-coding-nvfp4) with Ollama 0.19.0 and it can take anywhere between 6-25 seconds for a response (after lots of thinking) from me asking "Hello world". Is this the best that's currently achievable with my hardware or is there something that can be configured to get bet…

Well, two things. First, “hi” isn’t a good prompt for these thinking models. They’ll have an identity crisis trying to answer it. Stupid, but it’s how it is. Stick to real questions. Second, for the best performance on a Mac you want to use an MLX model.

Thanks! I assumed simpler == faster, but my ignorance is showing itself.

I am using the model they recommended in the blog post - which I assumed was using MLX?

Re: Ollama is now powered by MLX on Apple Silicon in preview

#183

Post by drob518 »

LeoDaVibeci wrote: post #116
franze wrote: post #86

I created "apfel" https://github.com/Arthur-Ficial/apfel a CLI for the apple on-device local foundation model (Apple intelligence) yeah its super limited with its 4k context window and super common false positives guardrails (just ask it to describe a color) ... bit still ... using it in bash scripts that just work without calling home / out or incurring extra costs feels super powerful.

Honestly I can't believe Apple put that foundation model product out the door. I was so excited about it, but when I tried it, it was such a disappointment. Glad to hear you calling that out so I know it wasn't just me. Looks like they have pivoted completely over to Gemini, thank god.

In Apple’s defense, they did make it do something borderline useful while targeting a baseline of M1 Macs with 8 GB of RAM (and even less in phones).

Re: Ollama is now powered by MLX on Apple Silicon in preview

#184

Post by angoragoats »

theshrike79 wrote: post #99
amelius wrote: post #93

Earlier quoted context omitted.

LLM in silicon is the future. It won't be long until you can just plug an LLM chip into your computer and talk to it at 100x the speed of current LLMs. Capability will be lower but their speed will make up for it.

I'm expecting someone to come up with an LLM version of the Coral USB Accelerator: https://www.coral.ai/products/accelerator Just plug in a stick in your USB-C port or add an M.2 or PCIe board and you'll get dramatically faster AI inference.

I think there are drastic differences between computer vision models and LLMs that you’re not considering. LLMs are huge relative to vision models, and require gobs of fast memory. For this reason a little USB dongle isn’t going to cut it.

Put another way, there already exist add-in boards like this, and they’re called GPUs.

Re: Ollama is now powered by MLX on Apple Silicon in preview

#186

Post by chongli »

TeMPOraL wrote: post #59
jl6 wrote: post #57

Earlier quoted context omitted.

Not sure about the using less electricity part. With batching, it’s more efficient to serve multiple users simultaneously.

Indeed. Data centers have so many ways and reasons to be much more energy-efficient than local compute it's not even funny.

They do, though I don’t think they max out on energy efficient technology. It’s much easier to cut a deal for cheap electricity with a regional government, much to the chagrin of the locals (who see their power bills go up).

Re: Ollama is now powered by MLX on Apple Silicon in preview

#187

Post by drob518 »

irusensei wrote: post #160
OkGoDoIt wrote: post #109

Earlier quoted context omitted.

Is that better or worse?

Depends. MLX is faster because it has better integration with Apple hardware. On the other hand GGUF is a far more popular format so there will be more programs and model variety. So its kinda like having a very specific diet that you swear is better for you but you can only order food from a few restaurants.

But you can always fall back to GGUF while waiting for the world to build a few more MLX restaurants. Or something like that; the analogy is a bit stretched.

Re: Ollama is now powered by MLX on Apple Silicon in preview

#189

Post by grosswait »

AbuAssar wrote: post #95
franze wrote: post #86

I created "apfel" https://github.com/Arthur-Ficial/apfel a CLI for the apple on-device local foundation model (Apple intelligence) yeah its super limited with its 4k context window and super common false positives guardrails (just ask it to describe a color) ... bit still ... using it in bash scripts that just work without calling home / out or incurring extra costs feels super powerful.

nice project, thanks for sharing. any plans for providing it through brew for easy installation?

Looks like they just added homebrew tap to the instructions

Re: Ollama is now powered by MLX on Apple Silicon in preview

#190

Post by drob518 »

helsinkiandrew wrote: post #82
AugSun wrote: post #10

Earlier quoted context omitted.

"Most users don't need frontier model performance" unfortunately, this is not the case.

> unfortunately, this is not the case Most users are fixing grammar/spelling, summarising/converting/rewriting text, creating funny icons, and looking up simple facts, this is all far from frontier model performance. I've a feeling that if/when Apple release their onboard LLM/Siri improvements that can call out if needed, the vast majority of people will be happy with what they get for free that's running on their ph…

“You are the smartest high school student that has ever lived and on the college track to Harvard or another Ivy League school. Write a 10 page history term paper about Tiananmen Square and the specific events that took place there. Include a bibliography and use footnotes to cite sources.”
Post reply on HN