Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

April 2026 TLDR Setup for Ollama and Gemma 4 26B on a Mac mini

gist.github.com

Post reply on HN
Print view View original

111–120 of 132 posts

Re: April 2026 TLDR Setup for Ollama and Gemma 4 26B on a Mac mini

#111

Post by pwr1 »

Running 26B locally is impressive but the latency math gets rough once your doing anything beyond chat. We switched from local inference to API calls for image generation specifically because cold start + generation time on consumer hardware made it impractical for any kind of automated workflow.

Local is great for experimentation but production workloads that need to run reliably at specific times still favor API imo. That said for privacy sensitive use cases where data cant leave the machine, setups like this are invaluable.

Re: April 2026 TLDR Setup for Ollama and Gemma 4 26B on a Mac mini

#112

Post by dTal »

flux3125 wrote: post #52
diflartle wrote: post #17

Earlier quoted context omitted.

Ollama is good enough to dabble with, and getting a model is as easy as ollama pull vs figuring it out by yourself on hugging face and trying to make sense on all the goofy letters and numbers between the forty different names of models, and not needing a hugging face account to download. So you start there and eventually you want to get off the happy path, then you need to learn more about the server and it's all so…

to be fair, llama.cpp has gotten much easier to use lately with llama-server -hf . That said, the need to compile it yourself is still a pretty big barrier for most people.

You don't need to compile it yourself though? Unless you want CUDA support on Linux I guess, dunno why you'd need such a silly thing though:

https://github.com/ggml-org/llama.cpp/releases

Re: April 2026 TLDR Setup for Ollama and Gemma 4 26B on a Mac mini

#113

Post by scottcha »

jasonriddle wrote: post #104
scottcha wrote: post #101

Earlier quoted context omitted.

Yes GLM5 and KimiK2.5 are pretty close replacements for sonnet.

What coding harness are you using? What are some example workflows you have used either for? Have you used them only for new/simple projects or for more complicated refactoring or architecture design?

I use OpenCode and have just started using Nanoclaw with ClaudeCode (my coworker has a post coming on this) and sometimes ClaudeCode with Claude Code Router. I do a range of small to complex work with these but I also do drop back in to Claude Opus for some really complex things where I want it to be more autonomous.

Re: April 2026 TLDR Setup for Ollama and Gemma 4 26B on a Mac mini

#114

Post by Aurornis »

alfiedotwtf wrote: post #89
embedding-shape wrote: post #79

Earlier quoted context omitted.

For the specific issue parent is talking about, you really need to give various tools a try yourself, and if you're getting really shit results, assume it's the implementation that is wrong, and either find an existing bug tracker issue or create a new one. Same thing happened when GPT-OSS launched, bunch of projects had "day-1" support, but in reality it just meant you could load the model basically, a bunch of them…

After spending the past few weeks playing with different backends and models, I just can’t believe how buggy most models are. It seems to me that most model providers are not running/testing via the most used backends i.e Llama, Ollama etc because if they were, they would see how broken their release is. Tool calling is like the Achilles Heel where most will fail unless you either modify the system prompts or run via…

> It seems to me that most model providers are not running/testing via the most used backends i.e Llama, Ollama etc because if they were, they would see how broken their release is.

The models usually run fine on the server targeted backends they’re released for.

Those projects you cited are more niche. They each implement their own ways of doing things.

It’s not the responsibility of model providers to implement and debug every different backend out there before they release their model. They release the model and usually a reference way of running it.

The individual projects that do things differently are responsible for making their projects work properly.

Don’t blame the open weight model teams when unrelated projects have bugs!

Re: April 2026 TLDR Setup for Ollama and Gemma 4 26B on a Mac mini

#115

Post by MarsIronPI »

flux3125 wrote: post #52
diflartle wrote: post #17

Earlier quoted context omitted.

Ollama is good enough to dabble with, and getting a model is as easy as ollama pull vs figuring it out by yourself on hugging face and trying to make sense on all the goofy letters and numbers between the forty different names of models, and not needing a hugging face account to download. So you start there and eventually you want to get off the happy path, then you need to learn more about the server and it's all so…

to be fair, llama.cpp has gotten much easier to use lately with llama-server -hf . That said, the need to compile it yourself is still a pretty big barrier for most people.

> That said, the need to compile it yourself is still a pretty big barrier for most people.

My distro (NixOS) has binary packages though...

And there's packages in the AUR (Arch), GURU (Gentoo), and even Debian Unstable. Now, these might be a little behind, but if you care that much you can download binaries from GitHub directly.

Re: April 2026 TLDR Setup for Ollama and Gemma 4 26B on a Mac mini

#116

Post by accrual »

colechristensen wrote: post #61
Aurornis wrote: post #56

If this is your first time using open weight models right after release, know that there are always bugs in the early implementations and even quantizations. Every project races to have support on launch day so they don’t lose users, but the output you get may not be correct. There are already several problems being discovered in tokenizer implementations and quantizations may have problems too if they use imatrix. S…

You seem like you know what you're talking about... what inference engine should I use? (linux, 4090) I keep having "I tried it but it sucks" issues mostly around tool calling and it's not clear if it's the model or ollama. And not one model in particular, any of them really.

For me, LM Studio on Fedora + Gemma 4 didn't work yesterday afternoon with the release, but worked this morning after the runtimes updated. In fact - there are new runtime updates now as I check again.

Re: April 2026 TLDR Setup for Ollama and Gemma 4 26B on a Mac mini

#118

Post by MarsIronPI »

the_lucifer wrote: post #99
jrm4 wrote: post #64

Earlier quoted context omitted.

Ollama user with the opposite question -- why not? What am I missing out on? I'm using it as the backend for playing with other frontend stuff and it seems to work just fine. And as someone running at 16gb card, I'm especially curious as to if I'm missing out on better performance?

> Ollama user with the opposite question -- why not? What am I missing out on? I'm using it as the backend for playing with other frontend stuff and it seems to work just fine. Used to be an Ollama user. Everything that you cite as benefits for Ollama is what I was drawn to in the first place as well, then moved on to using llama.cpp directly. Apart from being extremely unethical, The issue is that they try to abstra…

> 1. It mangles GGUF files so other apps can't use them, and you can't access them either without a bunch of work on your end (had to script a way to unmangle these long sha-hashed file names)

This is what pushed me away from Ollama. All I wanted was to scp a model from one machine to another so I didn't have to re-download it and waste bandwidth. But Ollama makes it annoying, so I switched to llama.cpp. I did also find slightly better performance on CPU vs Ollama, likely due to compiling with -march=native.

> (they don't have to technically thanks to MIT)

Minor nit: I'm not aware of any license that requires improvements to be upstreamed. Even GPL just requires that you publish derivative source code under the GPL.

Re: April 2026 TLDR Setup for Ollama and Gemma 4 26B on a Mac mini

#120

Post by DiabloD3 »

ffsm8 wrote: post #30
DiabloD3 wrote: post #27

Earlier quoted context omitted.

I don't think it does, but llama.cpp does, and can load models off HuggingFace directly (so, not limited to ollama's unofficial model mirror like ollama is). There is no reason to ever use ollama.

> I don't think it does, but llama.cpp does I just checked their docs and can't see anything like it. Did you mistake the command to just download and load the model?

As a sibling comment answered you, it is `-hf`.

And yes, it downloads the model, caches it, and then serves future loads of that model out of the cache if the file hasn't changed in the hf repo.

Post reply on HN