Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Ollama is now powered by MLX on Apple Silicon in preview

ollama.com

Post reply on HN
Print view View original

141–150 of 384 posts

Re: Ollama is now powered by MLX on Apple Silicon in preview

#141

Post by eigenspace »

theshrike79 wrote: post #118
harel wrote: post #89

What would be the non Mac computer to run these models locally at the same performance profile? Any similar linux ARM based computers that can reach the same level?

Framework Desktop is the closest one with the MAX 385/395 chip. It's mostly about the memory being fast enough rather than just CPU/GPU oomph. The 64GB model is 2240€ base and the 128GB is 3069€ base + all the stuff you need to add to make it an actual computer. As a comparison the 64GB Mac Mini is 2499€ here and a 128GB Mac Studio is 4274€.

Note though that that a MAX 395 has half the memory bandwidth of a M4 Max chip, and the memory bandwidth is going to be the biggest limiting factor, so you'll likely be getting around half the tokens/second with that Framework Desktop.

Re: Ollama is now powered by MLX on Apple Silicon in preview

#142

Post by WesolyKubeczek »

troad wrote: post #51
babblingfish wrote: post #2

LLMs on device is the future. It's more secure and solves the problem of too much demand for inference compared to data center supply, it also would use less electricity. It's just a matter of getting the performance good enough. Most users don't need frontier model performance.

I very recently installed llama.cpp on my consumer-grade M4 MBP, and I've been having loads of fun poking and prodding the local models. There's now a ChatGPT style interface baked into llama.cpp, which is very handy for quick experimentation. (I'm not entirely sure what Ollama would get me that llama.cpp doesn't, happy to hear suggestions!) There are some surprisingly decent models that happily fit even into a mere…

Cool, I always wanted to invade Belgium. Maybe if my plan is good, I could run a successful gofundme?

Re: Ollama is now powered by MLX on Apple Silicon in preview

#143

Post by zozbot234 »

jillesvangurp wrote: post #140
amelius wrote: post #93

Earlier quoted context omitted.

LLM in silicon is the future. It won't be long until you can just plug an LLM chip into your computer and talk to it at 100x the speed of current LLMs. Capability will be lower but their speed will make up for it.

You can always delegate sub agents to cloud based infrastructure for things that need more intelligence. But the future indeed is to keep the core interaction loop on the local device always ready for your input. A lot of stuff that we ask of these models isn't all that hard. Summarize this, parse that, call this tool, look that up, etc. 99.999% really isn't about implementing complex algorithms, solving important ma…

> This will drive an overdue increase in memory size of phones and laptops.

DRAM costs are still skyrocketing, so no, I don't think so. It's more likely that we'll bring back wear-resistant persistent memory as formerly seen with Intel Optane.

Re: Ollama is now powered by MLX on Apple Silicon in preview

#144

Post by cyanydeez »

AugSun wrote: post #10
babblingfish wrote: post #2

LLMs on device is the future. It's more secure and solves the problem of too much demand for inference compared to data center supply, it also would use less electricity. It's just a matter of getting the performance good enough. Most users don't need frontier model performance.

"Most users don't need frontier model performance" unfortunately, this is not the case.

eh, its weird how thetech world wants to build trillions of data centers for...what, escapingthe permanent underclass?

I think what "need" youspeak of is a bit of a colored statement.

Re: Ollama is now powered by MLX on Apple Silicon in preview

#145

Post by 3yr-i-frew-up »

franze wrote: post #86

I created "apfel" https://github.com/Arthur-Ficial/apfel a CLI for the apple on-device local foundation model (Apple intelligence) yeah its super limited with its 4k context window and super common false positives guardrails (just ask it to describe a color) ... bit still ... using it in bash scripts that just work without calling home / out or incurring extra costs feels super powerful.

[dead]

Re: Ollama is now powered by MLX on Apple Silicon in preview

#146

Post by dwayne_dibley »

babblingfish wrote: post #2

LLMs on device is the future. It's more secure and solves the problem of too much demand for inference compared to data center supply, it also would use less electricity. It's just a matter of getting the performance good enough. Most users don't need frontier model performance.

This might be how Apple will start to see even more sales, the M series processors are so far ahead of anything else, local LLMs could be their main selling point.

Re: Ollama is now powered by MLX on Apple Silicon in preview

#147

Post by vorticalbox »

robotswantdata wrote: post #83

Why are people still using Ollama? Serious. Lemonade or even llama.cpp are much better optimised and arguably just as easy to use.

i like ollama, mostly because the cli is pretty nice. its desktop app has stupid choices like if a model can support tools then the ui should give me the "search" option but it only shows for cloud models.

i have ran lmstudio for a while but i don't really use local models that much other than to mess about.

Re: Ollama is now powered by MLX on Apple Silicon in preview

#148

Post by 3yr-i-frew-up »

babblingfish wrote: post #2

LLMs on device is the future. It's more secure and solves the problem of too much demand for inference compared to data center supply, it also would use less electricity. It's just a matter of getting the performance good enough. Most users don't need frontier model performance.

[dead]

Re: Ollama is now powered by MLX on Apple Silicon in preview

#149

Post by JumpCrisscross »

franze wrote: post #86

I created "apfel" https://github.com/Arthur-Ficial/apfel a CLI for the apple on-device local foundation model (Apple intelligence) yeah its super limited with its 4k context window and super common false positives guardrails (just ask it to describe a color) ... bit still ... using it in bash scripts that just work without calling home / out or incurring extra costs feels super powerful.

…is it a reference to apfelwein?

Re: Ollama is now powered by MLX on Apple Silicon in preview

#150

Post by zozbot234 »

vorticalbox wrote: post #147
robotswantdata wrote: post #83

Why are people still using Ollama? Serious. Lemonade or even llama.cpp are much better optimised and arguably just as easy to use.

i like ollama, mostly because the cli is pretty nice. its desktop app has stupid choices like if a model can support tools then the ui should give me the "search" option but it only shows for cloud models. i have ran lmstudio for a while but i don't really use local models that much other than to mess about.

You can also use OpenWebUI locally which should give you a nice friendly UX once you set it up.
Post reply on HN