Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Ollama is now powered by MLX on Apple Silicon in preview

ollama.com

Post reply on HN
Print view View original

191–200 of 384 posts

Re: Ollama is now powered by MLX on Apple Silicon in preview

#191

Post by pama »

dgb23 wrote: post #174
konschubert wrote: post #152

Earlier quoted context omitted.

I disagree with every sentence of this. > solves the problem of too much demand for inference False, it creates consumer demand for inference chips, which will be badly utilised. > also would use less electricity What makes you think that? (MAYBE you can save power on cooling. But not if the data center is close to a natural heat sink) > It's just a matter of getting the performance good enough. The performance limit…

> False, it creates consumer demand for inference chips, which will be badly utilised. I think the opposite is true. Local inference doesn't have to go over the wire and through a bunch of firewalls and what have you. The performance from just regular consumer hardware with local, smaller models is already decent. You're utilizing the hardware you already have. > The performance limitations are inherent to the limite…

Parallel inference on large compute scales in superlinear ways. There is no way to beat the reduction in memory transfers that a data-center inference model provides with hardware that fits at anything called a home. It is much more energy efficient to process huge batches of parallel requests compared to having one or a handful of queries running on an accelerator.

Re: Ollama is now powered by MLX on Apple Silicon in preview

#192

Post by franze »

woadwarrior01 wrote: post #139
AbuAssar wrote: post #95

Earlier quoted context omitted.

nice project, thanks for sharing. any plans for providing it through brew for easy installation?

There's a very similar afm CLI that can be installed via Homebrew. https://github.com/scouzi1966/maclocal-api

done

  brew tap Arthur-Ficial/tap
  brew install Arthur-Ficial/tap/apfel

Re: Ollama is now powered by MLX on Apple Silicon in preview

#193

Post by g947o »

babblingfish wrote: post #2

LLMs on device is the future. It's more secure and solves the problem of too much demand for inference compared to data center supply, it also would use less electricity. It's just a matter of getting the performance good enough. Most users don't need frontier model performance.

Have you spent more than 10 min actually running LLM on a local machine?

As it stands today, local LLMs don't work remotely as well as some people try to picture them, in almost every way -- speed, performance, cost, usability etc. The only upside is privacy.

Re: Ollama is now powered by MLX on Apple Silicon in preview

#195

Post by theshrike79 »

troad wrote: post #158
theshrike79 wrote: post #97

Earlier quoted context omitted.

Qwen3.5 has tool calling, so you can give it a wikipedia tool which it uses to know what happened in Tiananmen Square without issues =)

That's very cool! I think giving it some research tools might be a nifty thing to try next. This is a fairly new area for me, so pointers or suggestions are welcome, even basic ones. :) Worth adding that I had reasoning on for the Tiananmen question, so I could see the prep for the answer, and it had a pretty strong current of "This is a sensitive question to PRC authorities and I must not answer, or even hint at an…

Basically ask any coding agent to create you a simple tool-calling harness for a local model and it'll most likely one-shot it.

Getting the local weather using a free API like met.no is a good first tool to use.

Re: Ollama is now powered by MLX on Apple Silicon in preview

#196

Post by theshrike79 »

eigenspace wrote: post #141
theshrike79 wrote: post #118

Earlier quoted context omitted.

Framework Desktop is the closest one with the MAX 385/395 chip. It's mostly about the memory being fast enough rather than just CPU/GPU oomph. The 64GB model is 2240€ base and the 128GB is 3069€ base + all the stuff you need to add to make it an actual computer. As a comparison the 64GB Mac Mini is 2499€ here and a 128GB Mac Studio is 4274€.

Note though that that a MAX 395 has half the memory bandwidth of a M4 Max chip, and the memory bandwidth is going to be the biggest limiting factor, so you'll likely be getting around half the tokens/second with that Framework Desktop.

There's a reason why it's cheaper than the Mac equivalent and it's not all because of Apple's premium pricing =)

But it's still the easiest and cleanest way to get decent local AI speeds on a non-Mac.

Re: Ollama is now powered by MLX on Apple Silicon in preview

#197

Post by g947o »

rubymamis wrote: post #135
dial9-1 wrote: post #4

still waiting for the day I can comfortably run Claude Code with local llm's on MacOS with only 16gb of ram

Doesn't OpenCode supports local models?

You can, but the quality sucks.

Local LLMs don't make sense for most people compared to "cloud" services, even more so for coding.

Re: Ollama is now powered by MLX on Apple Silicon in preview

#198

Post by bearjaws »

dial9-1 wrote: post #4

still waiting for the day I can comfortably run Claude Code with local llm's on MacOS with only 16gb of ram

My super uninformed theory is that local LLM will trail foundation models by about 2 years for practical use.

For example right now a lot of work is being done on improving tool calling and agentic workflows, which tool calling was first popping up around end of 2023 for local LLMs.

This is putting aside the standard benchmarks which get "benchmaxxed" by local LLMs and show impressive numbers, but when used with OpenCode rarely meet expectations. In theory Qwen3.5-397B-A17B should be nearly a Sonnet 4.6 model but it is not.

Re: Ollama is now powered by MLX on Apple Silicon in preview

#199

Post by RALaBarge »

g947o wrote: post #193
babblingfish wrote: post #2

LLMs on device is the future. It's more secure and solves the problem of too much demand for inference compared to data center supply, it also would use less electricity. It's just a matter of getting the performance good enough. Most users don't need frontier model performance.

Have you spent more than 10 min actually running LLM on a local machine? As it stands today, local LLMs don't work remotely as well as some people try to picture them, in almost every way -- speed, performance, cost, usability etc. The only upside is privacy.

I agree with you in the sense that if you tried to take any model right now and cram it into an iphone, it wouldnt be a claude-level agent.

I run 32b agents locally on a big video card, and smaller ones in CPU, but the lack there isn't the logic or reasoning, it is the chain of tooling that Claude Code and other stacks have built in.

Doing a lot of testing recently with my own harness, you would not believe the quality improvement you can get from a smaller LLM with really good opening context.

Even Microsoft is working on 1-bit LLMs...it sucks right now, but what about in 5 years?

But the OP is correct -- everything will have an LLM on it eventually, much sooner than people who do not understand what is going on right now would ever believe is possible.

Re: Ollama is now powered by MLX on Apple Silicon in preview

#200

Post by amelius »

angoragoats wrote: post #184
theshrike79 wrote: post #99

Earlier quoted context omitted.

I'm expecting someone to come up with an LLM version of the Coral USB Accelerator: https://www.coral.ai/products/accelerator Just plug in a stick in your USB-C port or add an M.2 or PCIe board and you'll get dramatically faster AI inference.

I think there are drastic differences between computer vision models and LLMs that you’re not considering. LLMs are huge relative to vision models, and require gobs of fast memory. For this reason a little USB dongle isn’t going to cut it. Put another way, there already exist add-in boards like this, and they’re called GPUs.

GPUs are still software programmable.

An "LLM chip" does not need that and so can be much more efficient.

Post reply on HN