Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

I ran Gemma 4 as a local model in Codex CLI

blog.danielvaughan.com

Post reply on HN
Print view View original

71–80 of 123 posts

Re: I ran Gemma 4 as a local model in Codex CLI

#71

Post by hnrodey »

fortyseven wrote: post #43
logicallee wrote: post #35

Earlier quoted context omitted.

Thanks for sharing that. What kind of hardware are you running this on?

4090, 128gb of ram (long before you'd have to take out a loan). I'm fairly sure it would run just as fine on a 3090. Thanks to the settings suggestions in the article, I was able to squeeze in the 31b model. Still testing, but it's real tight in 24gb of vram. A bit slower, too, but usable. Not sure I'm seeing much of a quality boost yet, but I'm still testing.

Probably a silly/obvious suggestion but are you using onboard GPU for display out?

Re: I ran Gemma 4 as a local model in Codex CLI

#72

Post by smrtinsert »

tuzemec wrote: post #9

I'm currently experimenting with running google/gemma-4-26b-a4b with lm studio ( https://lmstudio.ai/ ) and Opencode on a M3 Ultra with 48Gb RAM. And it seems to be working. I had to increase the context size to 65536 so the prompts from Opencode would work, but no other problems so far. I tried running the same on an M3 Max with less memory, but couldn't increase the context size enough to be useful with Opencode. I…

gguf or mlx? edit, just tried a community mlx and lm studio said it didn't support loading it yet.

Re: I ran Gemma 4 as a local model in Codex CLI

#73

Post by bitwize »

I recently spun up Gemma 4 26B-A4B on my local box and pointed OpenCode at it, and it did reasonably well! My machine is 8 years old, though I had the foresight to double the RAM to 32 GiB before the RAMpocalypse, and I can get a little bit of GPU oomph but not a lot, not with a mere GTX 1070. So it's slow, and nowhere near frontier model quality, but it can generate reasonable code and is good for faffing with!

Re: I ran Gemma 4 as a local model in Codex CLI

#74

Post by cjbgkagh »

I've been playing with this for the last few days. The model is fast, pretty smart, and I am hitting the same tool use issues. This blog post is unusually pertinent. The model speed isn't an issue on my dual 4090s, the productivity is mainly limited by the intelligence (while high it's still not high enough for some tasks) and getting stuck in loops.

What I would like is for it to be able detect when these things happen and to "Phone a Friend" to a smarter model to ask for advice.

I'm definitely moving into agent orchestration territory where I'll have an number of agents constantly running and working on things as I am not the bottleneck. I'll have a mix of on-prem and AI providers.

My role now is less coder and more designer / manager / architect as agents readily go off in tangents and mess that they're not smart enough to get out of.

Re: I ran Gemma 4 as a local model in Codex CLI

#75

Post by StrLght »

segmondy wrote: post #57

"The reason I had not done this before is that local models could not call tools. " Rubbish, we have been calling tools locally for 2 years, and it's very false that gemma3 scored under 7% in tool calling. Hell, I was getting at least 75% tool calling with llama3.3

I was also surprised by this sentence. It sounds like this is the author's first attempt at running models locally.

Or maybe the author has been running heavily quantized small models all that time — Gemma 4 gguf he's using is Q4 and only 16 GB. In my experience quants like this tend to perform much worse.

Re: I ran Gemma 4 as a local model in Codex CLI

#77

Post by hmokiguess »

m348e912 wrote: post #67
hmokiguess wrote: post #61

Earlier quoted context omitted.

what do you mean with that, I’m not sure I understood what you said

AI models like gemma4 are available in different quant "sizes", think about it as an image available in various compression levels. The best image is the largest, takes up the most memory when loading, and while it is large and looks the best, it uses up much of your system resources. On the other end of the spectrum there is a smaller much more compressed version of that same image. It loads quickly, uses less resou…

Thank you!

Re: I ran Gemma 4 as a local model in Codex CLI

#80

Post by prmoustache »

OutOfHere wrote: post #38
dist-epoch wrote: post #32

Earlier quoted context omitted.

> Imagine I am putting together electrical infrastructure, and the model gives me bad advice, risking electrocution and/or a serious fire That's a weird demand from models. What next, "Imagine I'm doing brain surgery and the model gives me bad advice", "Imagine I'm a judge delivering a sentencing and the model gives me bad advice", ...

Requesting electrical advice is not a weird ask at all. If writing sophisticated code requires skill, then so does electrical work, and one doesn't require more or less skill than the other. I would expect that the top-ranked thinking models are wholly capable of offering correct advice on the topic. The issues arise more from the user's inability to input all applicable context which can affect the decision and outp…

> Requesting electrical advice is not a weird ask at all. If writing sophisticated code requires skill, then so does electrical work

Except with electrical stuff the unit test itself can put your life and others in danger.

Post reply on HN