Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Ollama is now powered by MLX on Apple Silicon in preview

ollama.com

Post reply on HN
Print view View original

361–370 of 384 posts

Re: Ollama is now powered by MLX on Apple Silicon in preview

#361

Post by Nevermark »

raw_anon_1111 wrote: post #355
Nevermark wrote: post #334

Earlier quoted context omitted.

An E2EE system (e.g. as offered by Apple iCloud). Or a terms of service guarantee. (e.g. Dropbox, Anthropic and 1000 other companies that partition sharable user content from non-support divisions.) > Would the button disable them from checking in and updating their profile? No.

When you post a check in, your relationship status, your pictures without setting your sharing preferences and update your profile - you are specifically doing with the intention to share. WhatsApp is E2E encrypted

I already gave you a storage/share E2EE example.

I suggest that when things keep going over your head, like they did here, just Google the topic.

And when people are kind enough to reply to your confusion, read with a little more care.

Re: Ollama is now powered by MLX on Apple Silicon in preview

#362

Post by raw_anon_1111 »

Nevermark wrote: post #361
raw_anon_1111 wrote: post #355

Earlier quoted context omitted.

When you post a check in, your relationship status, your pictures without setting your sharing preferences and update your profile - you are specifically doing with the intention to share. WhatsApp is E2E encrypted

I already gave you a storage/share E2EE example. I suggest that when things keep going over your head, like they did here, just Google the topic. And when people are kind enough to reply to your confusion, read with a little more care.

I am not confused at all.

You’re arguing that people care about their privacy when they are explicitly sharing private information above what is needed to participate in FB.

You are completely wrong and your argument is illogical. People may not know that FB is making a profile of you based on your behavior. But logically, if I add to my profile that my favorite site is “grandma-midget-porn.com” [1], that I care that people don’t know I like senior citizen midgets

[1] Please don’t let that be a real website.

Re: Ollama is now powered by MLX on Apple Silicon in preview

#363

Post by nsonha »

konschubert wrote: post #152
babblingfish wrote: post #2

LLMs on device is the future. It's more secure and solves the problem of too much demand for inference compared to data center supply, it also would use less electricity. It's just a matter of getting the performance good enough. Most users don't need frontier model performance.

I disagree with every sentence of this. > solves the problem of too much demand for inference False, it creates consumer demand for inference chips, which will be badly utilised. > also would use less electricity What makes you think that? (MAYBE you can save power on cooling. But not if the data center is close to a natural heat sink) > It's just a matter of getting the performance good enough. The performance limit…

"consumer demand for inference chips, which will be badly utilised"

why do you assume it will be badly utilised? Can't be worse than what we have now which is chips already badly utilised for windows' bloatware

Re: Ollama is now powered by MLX on Apple Silicon in preview

#364

Post by Nevermark »

raw_anon_1111 wrote: post #362
Nevermark wrote: post #361

Earlier quoted context omitted.

I already gave you a storage/share E2EE example. I suggest that when things keep going over your head, like they did here, just Google the topic. And when people are kind enough to reply to your confusion, read with a little more care.

I am not confused at all. You’re arguing that people care about their privacy when they are explicitly sharing private information above what is needed to participate in FB. You are completely wrong and your argument is illogical. People may not know that FB is making a profile of you based on your behavior. But logically, if I add to my profile that my favorite site is “grandma-midget-porn.com” [1], that I care that…

Please reread what i wrote from the start more carefully.

I am not claiming what percentage of people care or not. I made a valid point of what is evidence or not, for not caring.

You also responded to my E2EE reference without absorbing the example.

It isn’t a big deal. We can move on.

Re: Ollama is now powered by MLX on Apple Silicon in preview

#365

Post by jillesvangurp »

zozbot234 wrote: post #143
jillesvangurp wrote: post #140

Earlier quoted context omitted.

You can always delegate sub agents to cloud based infrastructure for things that need more intelligence. But the future indeed is to keep the core interaction loop on the local device always ready for your input. A lot of stuff that we ask of these models isn't all that hard. Summarize this, parse that, call this tool, look that up, etc. 99.999% really isn't about implementing complex algorithms, solving important ma…

> This will drive an overdue increase in memory size of phones and laptops. DRAM costs are still skyrocketing, so no, I don't think so. It's more likely that we'll bring back wear-resistant persistent memory as formerly seen with Intel Optane.

Standard pig cycle in economics. Production capacity eventually goes up to meet demand and prices come down again. RAM has been going through cycles like this for decades. People seem to have no memory whatsoever of previous cycles every time it happens. Just wait a few years for it to become cheap again.

Re: Ollama is now powered by MLX on Apple Silicon in preview

#366

Post by mistercheese »

dabinat wrote: post #130
harel wrote: post #89

What would be the non Mac computer to run these models locally at the same performance profile? Any similar linux ARM based computers that can reach the same level?

Intel’s doing interesting things with their Arc GPUs. They’re offering GPUs that aren’t super fast for gaming but are relatively low power and have a boatload of VRAM. The new B70 is half the retail price of a 5090 (probably more like 1/3rd or 1/4 of actual 5090 selling prices) but has the same amount of memory and half the TDP. So for the same price as a 5090 you could get several and use them together.

Is it feasible to run LLM inference comparably without CUDA or Rocm? How much of the cost performance goes away?

Re: Ollama is now powered by MLX on Apple Silicon in preview

#367

Post by solatic »

beepbooptheory wrote: post #221
corndoge wrote: post #216

Earlier quoted context omitted.

that part is the system prompt, the script is a function that takes a prompt describing a shell command as an argument

But it's gotta be just a joke right? Which is why all the examples are just classic things you do with bash/unix utilities? I'll just say, if not a joke, the bit is appreciated either way! "AI change to the home directory. Make it snappy!"

mv /Users/beepbooptheory /Users/snappy

Re: Ollama is now powered by MLX on Apple Silicon in preview

#368

Post by raw_anon_1111 »

Nevermark wrote: post #364
raw_anon_1111 wrote: post #362

Earlier quoted context omitted.

I am not confused at all. You’re arguing that people care about their privacy when they are explicitly sharing private information above what is needed to participate in FB. You are completely wrong and your argument is illogical. People may not know that FB is making a profile of you based on your behavior. But logically, if I add to my profile that my favorite site is “grandma-midget-porn.com” [1], that I care that…

Please reread what i wrote from the start more carefully. I am not claiming what percentage of people care or not. I made a valid point of what is evidence or not, for not caring. You also responded to my E2EE reference without absorbing the example. It isn’t a big deal. We can move on.

Because it is irrelevant to whether people are purposefully explicitly sharing their likes and dislikes, and other information to let FB know more about them.

Re: Ollama is now powered by MLX on Apple Silicon in preview

#369

Post by LuxBennu »

goldenarm wrote: post #115
LuxBennu wrote: post #7

Already running qwen 70b 4-bit on m2 max 96gb through llama.cpp and it's pretty solid for day to day stuff. The mlx switch is interesting because ollama was basically shelling out to llama.cpp on mac before, so native mlx should mean better memory handling on apple silicon. Curious to see how it compares on the bigger models vs the gguf path

How many tokens per second?

Roughly 8-12 token/s on generation depending on context length. Prompt processing is faster obviously. Haven't benchmarked it super carefully though, just eyeballing the llama.cpp output.

Re: Ollama is now powered by MLX on Apple Silicon in preview

#370

Post by LuxBennu »

yg1112 wrote: post #305
LuxBennu wrote: post #7

Already running qwen 70b 4-bit on m2 max 96gb through llama.cpp and it's pretty solid for day to day stuff. The mlx switch is interesting because ollama was basically shelling out to llama.cpp on mac before, so native mlx should mean better memory handling on apple silicon. Curious to see how it compares on the bigger models vs the gguf path

The key difference is that MLX's array model assumes unified memory from the ground up. llama.cpp's Metal backend works fine but carries abstractions from the discrete GPU world — explicit buffer synchronization, command buffer boundaries — that are unnecessary when CPU and GPU share the same address space. You'll notice the gap most at large context lengths where KV cache pressure is highest.

that tracks with what i've noticed practically. shorter prompts feel basically the same between llama.cpp metal and what i'd expect from native mlx, but once context gets longer the overhead starts showing up. would be interesting to see if ollama's mlx path actually handles kv cache differently under the hood or if it just skips the buffer sync layer
Post reply on HN