Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Show HN: Open-source engine running Gemma 4 26B in 2 GB RAM on any M-series Mac

github.com

Post reply on HN
Print view View original

201–210 of 382 posts

Re: Show HN: Open-source engine running Gemma 4 26B in 2 GB RAM on any M-series Mac

#201

Post by gitpusher42 »

dboreham wrote: post #198

I'm intrigued to try a slightly different experiment: Did LLMs arise because a) humanity created circuits so large and so fast and so easy to use in parallel that only then did it become possible to run an LLM, or b) because sufficient data useful for training was accumulated such that experiments in different neural network arrangements could be done to see what came out? My hunch is (b) and so I further wonder how…

There was an ai winter for very long time. The math for NNs was already here, but not enough compute/data

I saw a pretty cool project to run an llm on an esp32 device https://github.com/slvDev/esp32-ai

Re: Show HN: Open-source engine running Gemma 4 26B in 2 GB RAM on any M-series Mac

#202

Post by evilduck »

govping wrote: post #150
itsdesmond wrote: post #126

Earlier quoted context omitted.

The writing is made worse by a specific choice which the commentator identified. That’s actionable feedback.

how is it made worse in this case?

Because it's a strong signal of AI slop. Why put in more work than the "author" did?

If the author generated text that required no effort, and has no understanding of the contents of the material generated, and no self-awareness of their behavior and how the audience will receive it, it definitely doesn't warrant wasting a single second reading it.

Now, granted, maybe they did review it, maybe they did understand it, maybe they did know how it would be received and merely made a mistake, but how are we to know? It quacks like a duck.

Re: Show HN: Open-source engine running Gemma 4 26B in 2 GB RAM on any M-series Mac

#203

Post by jryle70 »

JSR_FDED wrote: post #145
andrehacker wrote: post #107

Earlier quoted context omitted.

Let my karma burn for saying this: Maybe it is time to let this go man. These comments are really the new incarnation of "grammar policing". (1) They don't add anything of value, did the author use an LLM to fix his prose but no useless slop was added in the process: who cares ? Is the article useless slop: fine, downvote it to oblivion. (1) For those not old enough to remember that wonderful practice please use your…

Why would you want to signal low effort for your writing and the related project?

It's not low effort. The author had to write in English, not their native, and then used LLM to polish it. The sentence itself conveyed a real point. They cared how their article came across. That's much more effort than the boring "Claude was here" comment that took a second to write but costs real energy to appear, and once again spurred a worthless debate.

Let it go FFS.

Re: Show HN: Open-source engine running Gemma 4 26B in 2 GB RAM on any M-series Mac

#204

Post by s0a »

IdontKnowRust wrote: post #196
Keyframe wrote: post #168

Earlier quoted context omitted.

I don't care who King Charles is every single time that's the trick and a multi-billion dollar question, how would an llm engine know that? it's an active research area how to cull the initial layer surface and do the optimal traversal path through the layers and it's a damn hard problem. It's definitely an area where a ton of performance is left on the table still.

This makes me think (by full ignorance) how quantum computing can be useful in this field of things reach a stable-ish point

they will not. quantum computers as understood today will never run an LLM. read = destroy. and no cloning. wanna reload the entire model for each token? preparing quantum states is slow by any standard.

Re: Show HN: Open-source engine running Gemma 4 26B in 2 GB RAM on any M-series Mac

#205

Post by rpdillon »

evilduck wrote: post #202
govping wrote: post #150

Earlier quoted context omitted.

how is it made worse in this case?

Because it's a strong signal of AI slop. Why put in more work than the "author" did? If the author generated text that required no effort, and has no understanding of the contents of the material generated, and no self-awareness of their behavior and how the audience will receive it, it definitely doesn't warrant wasting a single second reading it. Now, granted, maybe they did review it, maybe they did understand it,…

The term "AI slop" is thought-terminating. A more nuanced approach: read it and decide for yourself on merits, rather than vibes.

Re: Show HN: Open-source engine running Gemma 4 26B in 2 GB RAM on any M-series Mac

#206

Post by rpdillon »

latexr wrote: post #144
andrehacker wrote: post #107

Earlier quoted context omitted.

Let my karma burn for saying this: Maybe it is time to let this go man. These comments are really the new incarnation of "grammar policing". (1) They don't add anything of value, did the author use an LLM to fix his prose but no useless slop was added in the process: who cares ? Is the article useless slop: fine, downvote it to oblivion. (1) For those not old enough to remember that wonderful practice please use your…

> Is the article useless slop: fine, downvote it to oblivion. You can’t downvote submissions on HN, only flag them. Identifying when text was written by LLMs is a useful signal. Maybe you don’t like these repeated comments, but I’d bet the people making them hate even more that they feel they wasted their time reading it.

This attitude will just cause people to write the same thing with AI and then ask it to strip out all the tells. I found it works just fine, but then you push usage underground, making it harder to detect, which isn't in your best interest, assuming you want to be able to detect and avoid low-effort writing.

Re: Show HN: Open-source engine running Gemma 4 26B in 2 GB RAM on any M-series Mac

#207

Post by ComputerPerson »

Keyframe wrote: post #168
giancarlostoro wrote: post #92

Nice, I think this is the second time I see this here on HN, I always wondered why we need to shove the entire model into memory, I don't care who King Charles is every single time. It always felt as though we already figured out how to break up large files and parse them efficiently with very little memory. Frontier AI feels like its full of people who are brilliant at making models, but when it comes to scale and p…

I don't care who King Charles is every single time that's the trick and a multi-billion dollar question, how would an llm engine know that? it's an active research area how to cull the initial layer surface and do the optimal traversal path through the layers and it's a damn hard problem. It's definitely an area where a ton of performance is left on the table still.

Seems to me like there's a conflict of incentives in the current era of market capture.

Not to say the cost-cutting wouldn't be valuable; today's race is predominantly about the model's reasoning capacity or "how hard of a math problem can the model solve".

It'll be a nice day when research-oriented human capital gets redirected to things that benefit us layfolks's pockets more directly

Re: Show HN: Open-source engine running Gemma 4 26B in 2 GB RAM on any M-series Mac

#209

Post by bitwize »

0x20cowboy wrote: post #146
micromacrofoot wrote: post #121

Earlier quoted context omitted.

you're absolutely right

But here’s the thing nobody tells you, it’s a repository not a spaceship. Not a pizza, not a cow, but an undeniable disco boot. Let’s delve into this.

Now I've got the whole picture.

Re: Show HN: Open-source engine running Gemma 4 26B in 2 GB RAM on any M-series Mac

#210

Post by gitpusher42 »

a-dub wrote: post #50
gitpusher42 wrote: post #33

Earlier quoted context omitted.

My first version used plain `mmap`. On the 8 GB M2, a cold 3.36 MB expert took 10 ms with mmap and 2.8 ms with `pread`. The full simulation was 0.50tok/s for `mmap` vs 4 tok/s for `pread` With `mmap`, OS loads pages reactively as the model touches them. It doesn’t know which experts were selected or when their reads could overlap with GPU work And common weights still use mmap for simplicity So, I believe llama.cpp m…

for a given expert, do you have a sense for what the spatiotemporal access pattern looks like?

Yeah, I checked it. One expert is about a 3.36mb block. If a cache miss happens I read whole block with one pread.

And there is some reuse. ~41% selected again for the next token, ~57% within two. Each layer has its own experts, so no reuse between these layers.

Post reply on HN