still waiting for the day I can comfortably run Claude Code with local llm's on MacOS with only 16gb of ram
[dead]
Ollama is now powered by MLX on Apple Silicon in preview
151–160 of 384 posts
Re: Ollama is now powered by MLX on Apple Silicon in preview
#152LLMs on device is the future. It's more secure and solves the problem of too much demand for inference compared to data center supply, it also would use less electricity. It's just a matter of getting the performance good enough. Most users don't need frontier model performance.
I disagree with every sentence of this.
> solves the problem of too much demand for inference
False, it creates consumer demand for inference chips, which will be badly utilised.
> also would use less electricity
What makes you think that? (MAYBE you can save power on cooling. But not if the data center is close to a natural heat sink)
> It's just a matter of getting the performance good enough.
The performance limitations are inherent to the limited compute and memory.
> Most users don't need frontier model performance.
What makes you think that?
Re: Ollama is now powered by MLX on Apple Silicon in preview
#153LLMs on device is the future. It's more secure and solves the problem of too much demand for inference compared to data center supply, it also would use less electricity. It's just a matter of getting the performance good enough. Most users don't need frontier model performance.
> It's just a matter of getting the performance good enough. Who will pay for the ongoing development of (near-)SoTA local models? The good open-weight models are all developed by for-profit companies - you know how that story will end.
Apple via customers paying for the whole solution ( eg a laptop that can run decent local models )?
I think Apple had something in the region of 143 billion in revenue in the last quarter.
Not saying it will happen - just that there are a variety of business models out there and in the end it all depends on where consumers put their money.
Re: Ollama is now powered by MLX on Apple Silicon in preview
#154Earlier quoted context omitted.
I very recently installed llama.cpp on my consumer-grade M4 MBP, and I've been having loads of fun poking and prodding the local models. There's now a ChatGPT style interface baked into llama.cpp, which is very handy for quick experimentation. (I'm not entirely sure what Ollama would get me that llama.cpp doesn't, happy to hear suggestions!) There are some surprisingly decent models that happily fit even into a mere…
Cool, I always wanted to invade Belgium. Maybe if my plan is good, I could run a successful gofundme?
Hey, if Margaret Thatcher's son can give it a go, why not you? Believe in yourself and reach for those dreams. *sparkle emoji*
Re: Ollama is now powered by MLX on Apple Silicon in preview
#155Earlier quoted context omitted.
Yep, having a "stupid" central model with multiple tools is IMO the key to efficient agentic systems. It needs to be just smart enough to use the tools and distill the responses into something usable. And one of the tools can be "ask claude/codex/gemini" so the local model itself doesn't actually need to do much.
> Yep, having a "stupid" central model with multiple tools is IMO the key to efficient agentic systems. That doesn't fix the "you don't know what you don't know" problem which is huge with smaller models. A bigger model with more world knowledge really is a lot smarter in practice, though at a huge cost in efficiency.
Ive always wondered where the inflection point lies between on the one hand trying to train the model on all kinds of data such as Wikipedia/encyclopedia, versus in the system prompt pointing to your local versions of those data sources, perhaps even through a search like api/tool.
Is there already some research or experimentation done into this area?
Re: Ollama is now powered by MLX on Apple Silicon in preview
#156Re: Ollama is now powered by MLX on Apple Silicon in preview
#157I created "apfel" https://github.com/Arthur-Ficial/apfel a CLI for the apple on-device local foundation model (Apple intelligence) yeah its super limited with its 4k context window and super common false positives guardrails (just ask it to describe a color) ... bit still ... using it in bash scripts that just work without calling home / out or incurring extra costs feels super powerful.
…is it a reference to apfelwein?
[deleted]
Re: Ollama is now powered by MLX on Apple Silicon in preview
#158Earlier quoted context omitted.
I very recently installed llama.cpp on my consumer-grade M4 MBP, and I've been having loads of fun poking and prodding the local models. There's now a ChatGPT style interface baked into llama.cpp, which is very handy for quick experimentation. (I'm not entirely sure what Ollama would get me that llama.cpp doesn't, happy to hear suggestions!) There are some surprisingly decent models that happily fit even into a mere…
Qwen3.5 has tool calling, so you can give it a wikipedia tool which it uses to know what happened in Tiananmen Square without issues =)
That's very cool! I think giving it some research tools might be a nifty thing to try next. This is a fairly new area for me, so pointers or suggestions are welcome, even basic ones. :)
Worth adding that I had reasoning on for the Tiananmen question, so I could see the prep for the answer, and it had a pretty strong current of "This is a sensitive question to PRC authorities and I must not answer, or even hint at an answer". I'm not sure if a research tool would be sufficient to overcome that censorship, though I guess I'll find out!
Re: Ollama is now powered by MLX on Apple Silicon in preview
#159Earlier quoted context omitted.
Have you used GPT instant or mini yourself? I think it’s pretty cynical to assume that this is “good enough for most people”, even if they don’t know the difference between that and better models.
Say more. Why do you think this?
They're awful and hallucinate a lot, I couldn't imagine using it even for prompts about TV shows, even less so for serious work. Repeating the question from the parent, have you tried those yourself? Even compared to ChatGPT Thinking, they're short of useless.
Re: Ollama is now powered by MLX on Apple Silicon in preview
#160Earlier quoted context omitted.
>Oh does llama.cpp use MLX or whatever? No. It runs on MacOS but uses Metal instead of MLX.
Is that better or worse?
Depends.
MLX is faster because it has better integration with Apple hardware. On the other hand GGUF is a far more popular format so there will be more programs and model variety.
So its kinda like having a very specific diet that you swear is better for you but you can only order food from a few restaurants.