Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Running Gemma 4 locally with LM Studio's new headless CLI and Claude Code

ai.georgeliu.com

Post reply on HN
Print view View original

81–90 of 121 posts

Re: Running Gemma 4 locally with LM Studio's new headless CLI and Claude Code

#81

Post by the_pwner224 »

mikkupikku wrote: post #78
zozbot234 wrote: post #35

Earlier quoted context omitted.

With unified memory, reading from RAM to GPU compute buffer is not that painful, and you can use partial RAM caching to minimize the impact of other kinds of swapping.

In practical terms, is this kind of architecture available to consumers except through Apple?

AMD Strix Halo. Available in the Framework desktop, various mini PCs, and the Asus Rog Flow Z13 "gaming tablet." The Z13 is still at $2700 for 128 GB which is an incredible deal with today's RAM prices.

There's also the Nvidia DGX Spark.

Re: Running Gemma 4 locally with LM Studio's new headless CLI and Claude Code

#82

Post by mjlee »

edinetdb wrote: post #60

[flagged]

I find MCP beneficial too, but do be aware of token usage. With a naive implementation MCP can use significantly more input tokens (and context) than equivalent skills would. With a handful of third party MCPs I’ve seen tens of thousands of tokens used before I’ve started anything.

Here’s an article from Anthropic explaining why, but it is 5 months old so perhaps it's irrelevant ancient history at this point.

https://www.anthropic.com/engineering/code-execution-with-mc...

Re: Running Gemma 4 locally with LM Studio's new headless CLI and Claude Code

#83

Post by d4rkp4ttern »

You can use llama.cpp server directly to serve local LLMs and use them in Claude Code or other CLI agents. I’ve collected full setup instructions for Gemma4 and other recent open-weight LLMs here, tested on my M1 Max 64 GB MacBook:

https://pchalasani.github.io/claude-code-tools/integrations/...

The 26BA4B is the most interesting to run on such hardware, and I get nearly double the token-gen speed (40 tok/s) compared to Qwen3.5 35BA3B. However the tau2 bench results[1] for this Gemma4 variant lag far behind the Qwen variant (68% vs 81%), so I don’t expect the former to do well on heavy agentic tool-heavy tasks:

[1] https://news.ycombinator.com/item?id=47616761

Re: Running Gemma 4 locally with LM Studio's new headless CLI and Claude Code

#84

Post by trvz »

alfiedotwtf wrote: post #68
pshirshov wrote: post #39

Earlier quoted context omitted.

Even tried gemma4:31b and gemma4:31b with 128k context (I have 72GiB VRAM). Nothing. I'm cursed I guess. That's ollama-rocm if that matters (I had weird bugs on Vulkan, maybe gemma misbehaves on radeons somehow?..). UPD: tried ollama-vulkan. It works, gemma4:31b-it-q8_0 with 64k context!

The default context is 128k for the smaller Gemma 4’s and 256k for the bigger ones, so you’re cutting off context and it doesn’t know how to continue. Bump it to native (or -c 0 may work too)

No, the default context in Ollama varies by the memory available: https://docs.ollama.com/context-length

Re: Running Gemma 4 locally with LM Studio's new headless CLI and Claude Code

#86

Post by peder »

d4rkp4ttern wrote: post #83

You can use llama.cpp server directly to serve local LLMs and use them in Claude Code or other CLI agents. I’ve collected full setup instructions for Gemma4 and other recent open-weight LLMs here, tested on my M1 Max 64 GB MacBook: https://pchalasani.github.io/claude-code-tools/integrations/... The 26BA4B is the most interesting to run on such hardware, and I get nearly double the token-gen speed (40 tok/s) compared…

Did you have any Anthropic vs OpenAI specification issues with Claude Code? I have been using mlx_vlm and vMLX and I get 400 Bad Request errors from Claude Code. Presumably you're not seeing those issues with llama-server ?

Re: Running Gemma 4 locally with LM Studio's new headless CLI and Claude Code

#87

Post by tatrions »

d4rkp4ttern wrote: post #83

You can use llama.cpp server directly to serve local LLMs and use them in Claude Code or other CLI agents. I’ve collected full setup instructions for Gemma4 and other recent open-weight LLMs here, tested on my M1 Max 64 GB MacBook: https://pchalasani.github.io/claude-code-tools/integrations/... The 26BA4B is the most interesting to run on such hardware, and I get nearly double the token-gen speed (40 tok/s) compared…

[flagged]

Re: Running Gemma 4 locally with LM Studio's new headless CLI and Claude Code

#88

Post by pshirshov »

nl wrote: post #62
pshirshov wrote: post #40

Earlier quoted context omitted.

Only for chat sessions, not for agentic coding. It's just too slow to be practical (10 minutes to answer a simple question about a 2k LoC project - and that's with a 5070 addon card).

Doesn't the framework desktop have a Ryzen 395 AI? That's a unified memory architecture like the Macs.

Ah, forgot to add, it's not really "unified" you have to explicitly specify your allocations. You may have a reasonably good 48gb chunk assigned to the GPU, but that DDR5 is 5-10 times slower than GDDR/HBM and the GPU itself isn't stellar.

So, framework laptops are great for chatting but nearly useless in agentic coding.

My Radeon W7900 answers a question ("what is this project") in 2 minutes, it takes my Framework 16 with 5070 addon around 11 minutes without the addon - around 23 (qwen 3.5 27b, claude code)

Post reply on HN