Earlier quoted context omitted.
The analysis has come back and the result is clear — the smoking gun is the belt-and-suspenders.
Do you want my honest, load-bearing opinion?
Say the word and I'll build it.
Show HN: Open-source engine running Gemma 4 26B in 2 GB RAM on any M-series Mac
191–200 of 382 posts
Re: Show HN: Open-source engine running Gemma 4 26B in 2 GB RAM on any M-series Mac
#192Earlier quoted context omitted.
But here’s the thing nobody tells you, it’s a repository not a spaceship. Not a pizza, not a cow, but an undeniable disco boot. Let’s delve into this.
The analysis has come back and the result is clear — the smoking gun is the belt-and-suspenders.
I got hit with my first belt-and-suspenders by Kimi K3 this morning. I normally use GPT. Is that a Claude-ism?
Re: Show HN: Open-source engine running Gemma 4 26B in 2 GB RAM on any M-series Mac
#193Earlier quoted context omitted.
I disagree. I thought of it as a noble public service. But as with everything else on the internet buyer beware.
Agree. I see it as informative to newcomers as to what they should do themselves. This is how people learn.
Right. Dude even included the prompt. Exemplary for this community I would hope.
Re: Show HN: Open-source engine running Gemma 4 26B in 2 GB RAM on any M-series Mac
#194Would it be possible to use this for kimi k3? what are the limitations
not with this engine. Kimi is a very different model. You can try to check HN later, I believe someone will build engine for this model for edge devices
Re: Show HN: Open-source engine running Gemma 4 26B in 2 GB RAM on any M-series Mac
#195Earlier quoted context omitted.
The analysis has come back and the result is clear — the smoking gun is the belt-and-suspenders.
I got hit with my first belt-and-suspenders by Kimi K3 this morning. I normally use GPT. Is that a Claude-ism?
Your observation is sharp. The honest answer: Yes.
Re: Show HN: Open-source engine running Gemma 4 26B in 2 GB RAM on any M-series Mac
#196Nice, I think this is the second time I see this here on HN, I always wondered why we need to shove the entire model into memory, I don't care who King Charles is every single time. It always felt as though we already figured out how to break up large files and parse them efficiently with very little memory. Frontier AI feels like its full of people who are brilliant at making models, but when it comes to scale and p…
I don't care who King Charles is every single time that's the trick and a multi-billion dollar question, how would an llm engine know that? it's an active research area how to cull the initial layer surface and do the optimal traversal path through the layers and it's a damn hard problem. It's definitely an area where a ton of performance is left on the table still.
This makes me think (by full ignorance) how quantum computing can be useful in this field of things reach a stable-ish point
Re: Show HN: Open-source engine running Gemma 4 26B in 2 GB RAM on any M-series Mac
#197You’re a mad man - thank you! Do I understand correctly that Ollama doesnt do that, and that’s why responses hang forever on a M3 running the same model through Ollama?
Thank you!
afaik ollama relies on llama.cpp and mmap. mmap loads pages on demand and doesn't use the same explicit cache or parallel reads like my engine. Most likely ollama/llama.cpp will be way slower in this case
Re: Show HN: Open-source engine running Gemma 4 26B in 2 GB RAM on any M-series Mac
#198I'm intrigued to try a slightly different experiment:
Did LLMs arise because a) humanity created circuits so large and so fast and so easy to use in parallel that only then did it become possible to run an LLM, or b) because sufficient data useful for training was accumulated such that experiments in different neural network arrangements could be done to see what came out?
My hunch is (b) and so I further wonder how far back in time could we have made a usable LLM if we had only known to try? E.g. can you run any sort of LLM on a VAX 11/780?
Re: Show HN: Open-source engine running Gemma 4 26B in 2 GB RAM on any M-series Mac
#199Earlier quoted context omitted.
Is there a particular quant of DS v4 Flash you'd recommend that works on 64GB machines? None of the antirez versions on HF look small enough? Also, FWIW, I've been experimenting with Laguna-S-2.1. It runs reasonably quickly (llama.cpp, IQ2_M quant) but the outputs so far aren't impressive, and it gets stuck and perseverates. Very subjectively, at that level of quantisation, it seems to perform worse than Qwen 3.6 27B…
For a dense model this would be a limitation, but not all of a MoE model needs to be in memory, but the largest part of a MoE are the routed experts. Some parts are needed to generated every single token and these really should fit in memory, but the router experts that are not neeed can rest on SSD and be read only if they are needed, so... you can run MoE models bigger than you memory, try the IQ2XXS. It should wor…
Thanks.
Agree on the sizing - selfishly, something like a 60B MoE would be great - fast on big machines, and a 4 or 5 bit quant should fit in 64GB and still work well.
Re: Show HN: Open-source engine running Gemma 4 26B in 2 GB RAM on any M-series Mac
#200[deleted]