Kind of interesting, really what experts do is sort/organise weights into categories that are optimal to work together. Seems like a lot of research could be done to extend this concept to group weights together for common inputs ahead of time to achieve the same purpose.
Show HN: Open-source engine running Gemma 4 26B in 2 GB RAM on any M-series Mac
211–220 of 382 posts
Re: Show HN: Open-source engine running Gemma 4 26B in 2 GB RAM on any M-series Mac
#212Thank you honestly
Re: Show HN: Open-source engine running Gemma 4 26B in 2 GB RAM on any M-series Mac
#213Earlier quoted context omitted.
Because it's a strong signal of AI slop. Why put in more work than the "author" did? If the author generated text that required no effort, and has no understanding of the contents of the material generated, and no self-awareness of their behavior and how the audience will receive it, it definitely doesn't warrant wasting a single second reading it. Now, granted, maybe they did review it, maybe they did understand it,…
The term "AI slop" is thought-terminating. A more nuanced approach: read it and decide for yourself on merits, rather than vibes.
Saying that something is thought terminating is thought terminating, it's the laziest "I win" bullshit approach ever. A more nuanced approach: don't produce slop and people won't dismiss it as lazy bullshit either.
Re: Show HN: Open-source engine running Gemma 4 26B in 2 GB RAM on any M-series Mac
#214I'm curious how your project compares to plain mmap! Because llama.cpp will already run 26B in 2GB of RAM if you really want to (mmap enabled, repacking disabled). It seems like the main difference is that your project synchronizes the SSD reads with inference activity, which you've presumably tuned to cause the least latency possible? Whereas the OS wouldn't care about any of that.
My first version used plain `mmap`. On the 8 GB M2, a cold 3.36 MB expert took 10 ms with mmap and 2.8 ms with `pread`. The full simulation was 0.50tok/s for `mmap` vs 4 tok/s for `pread` With `mmap`, OS loads pages reactively as the model touches them. It doesn’t know which experts were selected or when their reads could overlap with GPU work And common weights still use mmap for simplicity So, I believe llama.cpp m…
hm. in linux you have MAP_POPULATE which forces a prefetch where macos relies on page faults and lazy loading. if MADV_WILLNEED doesn't help, maybe readv to vector read directly or mmap+writev (write to a dummy fd, with an iovec for each page, hopefully resulting in a one-syscall-big-pagefault for mmap.) maybe also experiment with a loop that just reads one byte from each relevant page after mmap but before real computation?
Re: Show HN: Open-source engine running Gemma 4 26B in 2 GB RAM on any M-series Mac
#215How does this compare to DwarfStar4?
uh, I don't think it is possible to compare them. DwarfStar4 is for high end macs and a lot of ram. this project is more targeted to low end devices and "general use" Gemma4 model
Re: Show HN: Open-source engine running Gemma 4 26B in 2 GB RAM on any M-series Mac
#216Please explain what is useful about this repo to me as if I were a high school dropout. Is this like claude.ai but running on my own hardware? Does it need to be on the internet to be useful? Do I need CS skills to install and use it?
Re: Show HN: Open-source engine running Gemma 4 26B in 2 GB RAM on any M-series Mac
#217Earlier quoted context omitted.
> Is the article useless slop: fine, downvote it to oblivion. You can’t downvote submissions on HN, only flag them. Identifying when text was written by LLMs is a useful signal. Maybe you don’t like these repeated comments, but I’d bet the people making them hate even more that they feel they wasted their time reading it.
This attitude will just cause people to write the same thing with AI and then ask it to strip out all the tells. I found it works just fine, but then you push usage underground, making it harder to detect, which isn't in your best interest, assuming you want to be able to detect and avoid low-effort writing.
[deleted]
Re: Show HN: Open-source engine running Gemma 4 26B in 2 GB RAM on any M-series Mac
#218Earlier quoted context omitted.
Please don't do this. I know you mean well, but if you think you're providing a service here, you're not. This is not the same as posting an archive.today link to a paywalled article. This is not actually contributing anything to the discussion. Anyone who wants an LLM review can do so themselves. You have no idea if this is good output or slop. Nobody else knows if you even actually sent this through an LLM or not.
Friend, you have upvote / downvote with which you can signal to a person the value of their comment. Pontification from a four month old account at that.. nah.
We're not friends.
Re: Show HN: Open-source engine running Gemma 4 26B in 2 GB RAM on any M-series Mac
#219Are you running bare metal or docker? I had to go down to 2 bit on Gemma4 E2B model to run on 8gb on a Jetson
Quality and idempotency is great but it’s still not exactly fast… fast enough and works offline
Is this something that you can get running on Debian?
Re: Show HN: Open-source engine running Gemma 4 26B in 2 GB RAM on any M-series Mac
#220Thank you honestly
Thank you! If you can use it for your tasks I would be happy!