Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Show HN: Open-source engine running Gemma 4 26B in 2 GB RAM on any M-series Mac

github.com

Post reply on HN
Print view View original

311–320 of 382 posts

Re: Show HN: Open-source engine running Gemma 4 26B in 2 GB RAM on any M-series Mac

#311

Post by rufo »

musebox35 wrote: post #287
plandis wrote: post #167

Earlier quoted context omitted.

Isn’t this essentially what MoE partially solves with varying levels of accuracy?

Sadly no. Despite the name, the experts are not routed per concept or topic but per token. So for the same sentence you might activate multiple experts for different tokens. What it solves is the distributed training and inference problem. As long as each expert fits a single gpu, coordinating the model evaluation is much easier and it is faster. It does not buy as much for running on a single device though still les…

Apple’s new Foundation model for the 27 OS releases does some interesting things in exactly this area: https://machinelearning.apple.com/research/introducing-third...

Re: Show HN: Open-source engine running Gemma 4 26B in 2 GB RAM on any M-series Mac

#312

Post by sifarhub_com »

it looks good!!! now what I feel is showing map is fine but why to have confusing things around like rivers garden etc .It would be great if it can be kept minimum and simple so highlight would be food place and roads. also if possible try to integrate local delivery partner with transparent price of delivery charges. so user should not have to open multiple apps to see where is what price. I tried to login with email but it never reached to my mail the magic link.

Re: Show HN: Open-source engine running Gemma 4 26B in 2 GB RAM on any M-series Mac

#313

Post by gitpusher42 »

sudhirkhanger wrote: post #256

Is it possible to run Gemma or similar models on Raspberry Pi.

Yeah, must be possible. Not fast, but possible if you have enough ram. I think you can search online for projects, I think I saw something related

Re: Show HN: Open-source engine running Gemma 4 26B in 2 GB RAM on any M-series Mac

#314

Post by gitpusher42 »

tracker1 wrote: post #40

This is actually very similar to some ideas I've been having for a while... that having a smaller entry model that knows enough about "expert" models that themselves are smaller to hand work over to could be better/faster/lighter in terms of working through real problems vs the megalith ones we currently use. Highly distilled experts and coordination with a fallback mode to a larger model option.

afaik there is some research at this area. Also the new apple foundation model uses related idea. they process the whole prompt and based on prompt load required experts and use only these experts for generation. It doesn't require fitting full model into memory or per token ssd streaming

Re: Show HN: Open-source engine running Gemma 4 26B in 2 GB RAM on any M-series Mac

#315

Post by dTal »

rpdillon wrote: post #205
evilduck wrote: post #202

Earlier quoted context omitted.

Because it's a strong signal of AI slop. Why put in more work than the "author" did? If the author generated text that required no effort, and has no understanding of the contents of the material generated, and no self-awareness of their behavior and how the audience will receive it, it definitely doesn't warrant wasting a single second reading it. Now, granted, maybe they did review it, maybe they did understand it,…

The term "AI slop" is thought-terminating. A more nuanced approach: read it and decide for yourself on merits, rather than vibes.

That's like inhaling a virus to see if it's contagious. Reading text generated by a model tuned with RLHF is a dangerous pastime. It's all too easy to reach past a human's critical thinking and push the pleasure buttons. Uncanny valley is uncanny for a reason. As social monkeys we instinctively screech danger warnings at each other; AI slop gets the same treatment as an alligator pretending to be a log.

Re: Show HN: Open-source engine running Gemma 4 26B in 2 GB RAM on any M-series Mac

#316

Post by apothegm »

swiftcoder wrote: post #149
giancarlostoro wrote: post #92

Nice, I think this is the second time I see this here on HN, I always wondered why we need to shove the entire model into memory, I don't care who King Charles is every single time. It always felt as though we already figured out how to break up large files and parse them efficiently with very little memory. Frontier AI feels like its full of people who are brilliant at making models, but when it comes to scale and p…

> I don't care who King Charles is every single time Always curious when someone will figure out how we can elide most of the data from an LLM (but retain the logical ability). I don't actually need an LLM to have a very big internal knowledge base to be useful, so long as it can invoke a search tool...

That’s the thing. LLMs don’t have any logical ability. Only predictive ability. They’re not the same. And that’s why LLMs are a) unreliable and b) not a path to AGI.

Re: Show HN: Open-source engine running Gemma 4 26B in 2 GB RAM on any M-series Mac

#317

Post by gitpusher42 »

a-dub wrote: post #214
gitpusher42 wrote: post #33

Earlier quoted context omitted.

My first version used plain `mmap`. On the 8 GB M2, a cold 3.36 MB expert took 10 ms with mmap and 2.8 ms with `pread`. The full simulation was 0.50tok/s for `mmap` vs 4 tok/s for `pread` With `mmap`, OS loads pages reactively as the model touches them. It doesn’t know which experts were selected or when their reads could overlap with GPU work And common weights still use mmap for simplicity So, I believe llama.cpp m…

hm. in linux you have MAP_POPULATE which forces a prefetch where macos relies on page faults and lazy loading. if MADV_WILLNEED doesn't help, maybe readv to vector read directly or mmap+writev (write to a dummy fd, with an iovec for each page, hopefully resulting in a one-syscall-big-pagefault for mmap.) maybe also experiment with a loop that just reads one byte from each relevant page after mmap but before real comp…

uh, tried most of this

mmap benchmark did basically page touch experiment and cold reads were much slower, unfortunately (10ms vs 3ms)

I tried MADV_WILLNEED, F_RDADVISE and preadv. preadv reduced parallelism because requested experts are rarely adjacent in the file.

pread is still the fastest. And I think Flash-Moe got the same result too

Re: Show HN: Open-source engine running Gemma 4 26B in 2 GB RAM on any M-series Mac

#319

Post by anentropic »

gitpusher42 wrote: post #11
yakupov_bulat wrote: post #2

Wow, amazing! What if there is enough RAM to fully load the model? I assume in that case I shouldn’t use your engine.

It depends on the use case. I measured this exact model with a 4k context on the mlx engine. It runs at 75 tok/s on my M5 Mac Pro and using 14 GB of RAM. For my engine the same model uses 2 GB of RAM and produces 31–35 tok/s. The project is still experimental so performance may vary as it continues to improve. If you want to save around 12 GB of RAM for other tasks and you are ok with 35 tok/s (afaik it is roughly co…

can I vary the context length depending on RAM available?

Re: Show HN: Open-source engine running Gemma 4 26B in 2 GB RAM on any M-series Mac

#320

Post by apothegm »

woadwarrior01 wrote: post #84

> The measured result is a reference point, not a performance ceiling. Claude was here.

Okay, I’m the first to get annoyed at LLM-ese. But language requires the ability to say that something is A and not B. There are certain phrasings of that that are painfully Claude-esque, but ffs the one you quoted is the type of thing an actual human being is just as likely to say.
Post reply on HN