Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Ollama is now powered by MLX on Apple Silicon in preview

ollama.com

Post reply on HN
Print view View original

131–140 of 384 posts

Re: Ollama is now powered by MLX on Apple Silicon in preview

#131

Post by comboy »

nbenitezl wrote: post #127
babblingfish wrote: post #2

LLMs on device is the future. It's more secure and solves the problem of too much demand for inference compared to data center supply, it also would use less electricity. It's just a matter of getting the performance good enough. Most users don't need frontier model performance.

But when using it on the cloud a LLM can consult 50 websites, which is super fast for their datacenters as they are backbone of internet, instead you'll have to wait much more on your device to consult those websites before giving you the LLM response. Am i wrong?

As things stand today even when doing research tasks, time spent by model is >> than fetching websites. I don't see it changing any time soon, except when some deals happen behind the scenes where agents get to access CF guarded resources that normally get blocked from automated access.

Re: Ollama is now powered by MLX on Apple Silicon in preview

#132

Post by Const-me »

nbenitezl wrote: post #127
babblingfish wrote: post #2

LLMs on device is the future. It's more secure and solves the problem of too much demand for inference compared to data center supply, it also would use less electricity. It's just a matter of getting the performance good enough. Most users don't need frontier model performance.

But when using it on the cloud a LLM can consult 50 websites, which is super fast for their datacenters as they are backbone of internet, instead you'll have to wait much more on your device to consult those websites before giving you the LLM response. Am i wrong?

While data centres indeed have awesome internet connectivity, don’t forget the bandwidth is shared by all clients using a particular server.

If you have 100 mbit/sec internet connection at home, a computer in a data centre has 10 gbit/sec, but the server is serving 200 concurrent clients — your bandwidth is twice as fast.

Re: Ollama is now powered by MLX on Apple Silicon in preview

#133

Post by zozbot234 »

theshrike79 wrote: post #128
girvo wrote: post #121

Earlier quoted context omitted.

I'd recommend it too, because the knowledge cutoff of all the open weight Chinese models (M2.7, Qwen3.5, GLM-5 etc) is earlier than you'd think, so giving it web search (I use `ddgr` with a skill) helps a surprising amount

Yep, having a "stupid" central model with multiple tools is IMO the key to efficient agentic systems. It needs to be just smart enough to use the tools and distill the responses into something usable. And one of the tools can be "ask claude/codex/gemini" so the local model itself doesn't actually need to do much.

> Yep, having a "stupid" central model with multiple tools is IMO the key to efficient agentic systems.

That doesn't fix the "you don't know what you don't know" problem which is huge with smaller models. A bigger model with more world knowledge really is a lot smarter in practice, though at a huge cost in efficiency.

Re: Ollama is now powered by MLX on Apple Silicon in preview

#134

Post by rubymamis »

harel wrote: post #89

What would be the non Mac computer to run these models locally at the same performance profile? Any similar linux ARM based computers that can reach the same level?

I wonder if the Snapdragon X Elite already caught up with the Apple's M series in that regard - does anybody know?

Re: Ollama is now powered by MLX on Apple Silicon in preview

#137

Post by zozbot234 »

hu3 wrote: post #129
harel wrote: post #120

Earlier quoted context omitted.

It's odd no manufacturer jumped on this wagon to offer a competitive alternative.

Is there even enough market for this? These models are dumber and slower than API SoTA models and will always be. My time and sanity is much more expensive than insurance against any risk of sending my garbage code to companies worth hundreds of billions of dollars. For most, it's a downgrade to use local models in multiple fronts: total cost of ownership, software maintenance, electricity bill, losing performance on…

> These models are dumber and slower than API SoTA models and will always be.

Sure but you're paying per-token costs on the SoTA models that are roughly an order of magnitude higher than third-party inference on the locally available models. So when you account for per-token cost, the math skews the other way.

Re: Ollama is now powered by MLX on Apple Silicon in preview

#138

Post by theshrike79 »

zozbot234 wrote: post #133
theshrike79 wrote: post #128

Earlier quoted context omitted.

Yep, having a "stupid" central model with multiple tools is IMO the key to efficient agentic systems. It needs to be just smart enough to use the tools and distill the responses into something usable. And one of the tools can be "ask claude/codex/gemini" so the local model itself doesn't actually need to do much.

> Yep, having a "stupid" central model with multiple tools is IMO the key to efficient agentic systems. That doesn't fix the "you don't know what you don't know" problem which is huge with smaller models. A bigger model with more world knowledge really is a lot smarter in practice, though at a huge cost in efficiency.

That's the key, it just needs to be smart enough to 1) know it doesn't know and 2) "know a guy" as they say =) (call a tool for the exact information)

Picking a model that's juuust smart enough to know it doesn't know is the key.

Re: Ollama is now powered by MLX on Apple Silicon in preview

#139

Post by woadwarrior01 »

AbuAssar wrote: post #95
franze wrote: post #86

I created "apfel" https://github.com/Arthur-Ficial/apfel a CLI for the apple on-device local foundation model (Apple intelligence) yeah its super limited with its 4k context window and super common false positives guardrails (just ask it to describe a color) ... bit still ... using it in bash scripts that just work without calling home / out or incurring extra costs feels super powerful.

nice project, thanks for sharing. any plans for providing it through brew for easy installation?

There's a very similar afm CLI that can be installed via Homebrew.

https://github.com/scouzi1966/maclocal-api

Re: Ollama is now powered by MLX on Apple Silicon in preview

#140

Post by jillesvangurp »

amelius wrote: post #93
babblingfish wrote: post #2

LLMs on device is the future. It's more secure and solves the problem of too much demand for inference compared to data center supply, it also would use less electricity. It's just a matter of getting the performance good enough. Most users don't need frontier model performance.

LLM in silicon is the future. It won't be long until you can just plug an LLM chip into your computer and talk to it at 100x the speed of current LLMs. Capability will be lower but their speed will make up for it.

You can always delegate sub agents to cloud based infrastructure for things that need more intelligence. But the future indeed is to keep the core interaction loop on the local device always ready for your input.

A lot of stuff that we ask of these models isn't all that hard. Summarize this, parse that, call this tool, look that up, etc. 99.999% really isn't about implementing complex algorithms, solving important math problems, working your way through a benchmark of leet programming exercises, etc. You also really don't need these models to know everything. It's nice if it can hallucinate a decent answer to most questions. But the smarter way is to look up the right answer and then summarize it. Good enough goes a long way. Speed and latency are becoming a key selling point. You need enough capability locally to know when to escalate to something slower and more costly.

This will drive an overdue increase in memory size of phones and laptops. Laptops especially have been stuck at the same common base level of 8-16GB for about 15 years now. Apple still sells laptops with just 8GB (their new Neo). I had a 16 GB mac book pro in 2012. At the time that wasn't even that special. My current one has 48GB; enough for some of the nicer models. You can get as much as 256GB today.

Post reply on HN