Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Ollama is now powered by MLX on Apple Silicon in preview

ollama.com

Post reply on HN
Print view View original

351–360 of 384 posts

Re: Ollama is now powered by MLX on Apple Silicon in preview

#351

Post by Patrick_Devine »

duffyjp wrote: post #286
functional_dev wrote: post #235

Earlier quoted context omitted.

I did not know, that NVFP4 was handled at the silicon level... until I dug deeper here - https://vectree.io/c/llm-quantization-from-weights-to-bits-g...

I still don't think I understand it. I saw those nvfp4 models up by chance yesterday and tried them on my Linux PC with a 5060TI 16gb. Ollama refused to pull them saying they were macOS only. I assumed it was a meta-data bug and posted an issue, but apparently nvfp4 doesn't necessarily mean nvidia-fp4. https://github.com/ollama/ollama/issues/15149

They are nvidia-fp4 weights, but CUDA support isn't _quite_ ready yet, but we've got that cooking.

Re: Ollama is now powered by MLX on Apple Silicon in preview

#352

Post by irusensei »

drob518 wrote: post #187
irusensei wrote: post #160

Earlier quoted context omitted.

Depends. MLX is faster because it has better integration with Apple hardware. On the other hand GGUF is a far more popular format so there will be more programs and model variety. So its kinda like having a very specific diet that you swear is better for you but you can only order food from a few restaurants.

But you can always fall back to GGUF while waiting for the world to build a few more MLX restaurants. Or something like that; the analogy is a bit stretched.

Yeah I'm terrible with analogies.

Re: Ollama is now powered by MLX on Apple Silicon in preview

#354

Post by troad »

austinthetaco wrote: post #249
troad wrote: post #51

Earlier quoted context omitted.

I very recently installed llama.cpp on my consumer-grade M4 MBP, and I've been having loads of fun poking and prodding the local models. There's now a ChatGPT style interface baked into llama.cpp, which is very handy for quick experimentation. (I'm not entirely sure what Ollama would get me that llama.cpp doesn't, happy to hear suggestions!) There are some surprisingly decent models that happily fit even into a mere…

Have you played around with any of the Hermes models? they are supposed to be one of the best at non-refusal while keeping sane.

Interesting! Unfortunately, the smallest Hermes 4 model I can see is 14B, which would really strain the limits of my little laptop. The only way I might get acceptable performance would be to run it extremely quantised, but then I probably wouldn't see much improvement over the 9B Qwen.

Re: Ollama is now powered by MLX on Apple Silicon in preview

#355

Post by raw_anon_1111 »

Nevermark wrote: post #334
raw_anon_1111 wrote: post #333

Earlier quoted context omitted.

They are explicitly adding their information to FB why do they need a button to not share the information? Would the button disable them from checking in and updating their profile?

An E2EE system (e.g. as offered by Apple iCloud). Or a terms of service guarantee. (e.g. Dropbox, Anthropic and 1000 other companies that partition sharable user content from non-support divisions.) > Would the button disable them from checking in and updating their profile? No.

When you post a check in, your relationship status, your pictures without setting your sharing preferences and update your profile - you are specifically doing with the intention to share. WhatsApp is E2E encrypted

Re: Ollama is now powered by MLX on Apple Silicon in preview

#356

Post by angoragoats »

amelius wrote: post #200
angoragoats wrote: post #184

Earlier quoted context omitted.

I think there are drastic differences between computer vision models and LLMs that you’re not considering. LLMs are huge relative to vision models, and require gobs of fast memory. For this reason a little USB dongle isn’t going to cut it. Put another way, there already exist add-in boards like this, and they’re called GPUs.

GPUs are still software programmable. An "LLM chip" does not need that and so can be much more efficient.

Sure, but that’s somewhat orthogonal to the point I was making, which is that LLMs are huge in size. Even in the case of a custom “LLM chip,” you’ll need huge amounts of very fast storage of some sort (likely DRAM), which places constraints on the size, power consumption, and cost of such a device. This device, if it existed, would not in any way resemble the Coral TPU product that the GP was referencing; I think in fact it would be closer in size, price, and form factor to a GPU.

Re: Ollama is now powered by MLX on Apple Silicon in preview

#357

Post by g947o »

kylehotchkiss wrote: post #296
g947o wrote: post #193

Earlier quoted context omitted.

Have you spent more than 10 min actually running LLM on a local machine? As it stands today, local LLMs don't work remotely as well as some people try to picture them, in almost every way -- speed, performance, cost, usability etc. The only upside is privacy.

Yes. I've spent months running Qwen2.5-8B on my barebones 16gb ram M4 Mac mini to handle identifying sites from google search results. It has been rock solid. I'm not even running this MLX-powered improvement on it yet. Your idea of what people need from Local LLMs and others are different. Not everybody needs a /r/myboyfriendisai level performance.

You probably want to double check the comment I was responding to.

Re: Ollama is now powered by MLX on Apple Silicon in preview

#358

Post by troad »

theshrike79 wrote: post #195
troad wrote: post #158

Earlier quoted context omitted.

That's very cool! I think giving it some research tools might be a nifty thing to try next. This is a fairly new area for me, so pointers or suggestions are welcome, even basic ones. :) Worth adding that I had reasoning on for the Tiananmen question, so I could see the prep for the answer, and it had a pretty strong current of "This is a sensitive question to PRC authorities and I must not answer, or even hint at an…

Basically ask any coding agent to create you a simple tool-calling harness for a local model and it'll most likely one-shot it. Getting the local weather using a free API like met.no is a good first tool to use.

Thanks!

Re: Ollama is now powered by MLX on Apple Silicon in preview

#359

Post by esseph »

raw_anon_1111 wrote: post #348
esseph wrote: post #345

Earlier quoted context omitted.

> Again - anecdotes is not data. We have data. You have data showing growth in cloud, which I expect and don't disagree with. The data I come across shows this too! What I disagree with, from my own experiences and all the data I can seem to find online is that the growth rate in repatriation is MUCH higher than the growth in cloud. It has flipped over the last 3yr. US Enterprises, Fortune 100, especially. Also a lot…

Again, anecdotes. I have public company quarterly statements - you have unsourced quotes. You can quote Geico - I can quote Netflix. If on prem was really growing, I wouldn’t expect Intel to be in the shitter and I would expect Capex to be focused on Colo centers not cloud. Also when I searched for your quotation the very next paragraph was “ This trend does not represent a rejection of cloud computing. Organizations…

That doesn't dispute what I said, in fact it agrees with what I said. Read it again.

> You have data showing growth in cloud, which I expect and don't disagree with. The data I come across shows this too!

Re: Ollama is now powered by MLX on Apple Silicon in preview

#360

Post by selcuka »

lxgr wrote: post #70
selcuka wrote: post #34

Earlier quoted context omitted.

Any citations? Because that was my impression, too. I want frontier model performance for my coding assistant, but "most users" could do with smaller/faster models. ChatGPT free falls back to GPT-5.2 Mini after a few interactions.

Have you used GPT instant or mini yourself? I think it’s pretty cynical to assume that this is “good enough for most people”, even if they don’t know the difference between that and better models.

> I think it’s pretty cynical to assume that this is “good enough for most people”

It's a deduction, not an assumption. Obviously it's "good enough" for "most people". Otherwise nobody would be using the free version of ChatGPT today.

I pay for a Claude subscription, but even then I sometimes downgrade to Sonnet or even Haiku when I need a quick answer.

Post reply on HN