Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Running Gemma 4 locally with LM Studio's new headless CLI and Claude Code

ai.georgeliu.com

Post reply on HN
Print view View original

31–40 of 121 posts

Re: Running Gemma 4 locally with LM Studio's new headless CLI and Claude Code

#31

Post by blitzar »

mbesto wrote: post #25
unsnap_biceps wrote: post #6

Earlier quoted context omitted.

lm studio offers an Anthropic compatible local endpoint, so you can point Claude code at it and it'll use your local model for it's requests, however, I've had a lot of problems with LM Studio and Claude code losing it's place. It'll think for awhile, come up with a plan, start to do it and then just halt in the middle. I'll ask it to continue and it'll do a small change and get stuck again. Using ollama's api doesn'…

I don't get why I would use Claude Code when OpenCode, Cursor, Zed, etc. all exist, are "free" and work with virtually any llm. Seems like a weird use case unless I'm missing something.

previously I have found claude code to be just better than the alternatives, using large models or local. It is, however, closer now and not much excuse for the competition after the claude code leak. Personally, I will be giving this a go with OpenCode.

Re: Running Gemma 4 locally with LM Studio's new headless CLI and Claude Code

#32

Post by martinald »

charcircuit wrote: post #20
martinald wrote: post #11

Just FYI, MoE doesn't really save (V)RAM. You still need all weights loaded in memory, it just means you consult less per forward pass. So it improves tok/s but not vram usage.

You never need to have all weights in memory. You can swap them in from RAM, disk, the network, etc. MOE reduces the amount of data that will need to be swapped in for the next forward pass.

Yes you're right technically, but in reality you'd be swapping them the (vast?) majority in and out per inference request so would create an enormous bottleneck for the use case the author is using for.

Re: Running Gemma 4 locally with LM Studio's new headless CLI and Claude Code

#33

Post by blitzar »

jaggederest wrote: post #23
unsnap_biceps wrote: post #14

Earlier quoted context omitted.

I use CC at work, so I haven't explored other options. Is there a better one to use locally? I presumed they were all going to be pretty similar.

If you want to experiment with same-harness-different-models Opencode is classically the one to use. After their recent kerfluffle with Anthropic you'll have to use API pricing for opus/sonnet/haiku which makes it kind of a non-starter, but it lets you swap out any number of cloud or local models using e.g. ollama or z.ai or whatever backend provider you like. I'd rate their coding agent harness as slightly to signif…

I am hopeful the leaked claude code narrows the capability, perhaps even googles offering will be viable once they borrow some ideas from claude.

Re: Running Gemma 4 locally with LM Studio's new headless CLI and Claude Code

#34

Post by zozbot234 »

Yukonv wrote: post #24
IceWreck wrote: post #15

Earlier quoted context omitted.

It does if you use an inference engine where you can offload some of the experts from VRAM to CPU RAM. That means I can fit a 35 billion param MoE in let's say 12 GB VRAM GPU + 16 gigs of memory.

With that you are taking a significant performance penalty and become severely I/O bottlenecked. I've been able to stream Qwen3.5-397B-A17B from my M5 Max (12 GB/s SSD Read) using the Flash MoE technique at the brisk pace of 10 tokens per second. As tokens are generated different experts need to be consulted resulting in a lot of I/O churn. So while feasible it's only great for batch jobs not interactive usage.

10 tok/s is quite fine for chatting, though less so for interaction with agentic workloads. So the technique itself is still worthwhile for running a huge model locally.

Re: Running Gemma 4 locally with LM Studio's new headless CLI and Claude Code

#35

Post by zozbot234 »

martinald wrote: post #32
charcircuit wrote: post #20

Earlier quoted context omitted.

You never need to have all weights in memory. You can swap them in from RAM, disk, the network, etc. MOE reduces the amount of data that will need to be swapped in for the next forward pass.

Yes you're right technically, but in reality you'd be swapping them the (vast?) majority in and out per inference request so would create an enormous bottleneck for the use case the author is using for.

With unified memory, reading from RAM to GPU compute buffer is not that painful, and you can use partial RAM caching to minimize the impact of other kinds of swapping.

Re: Running Gemma 4 locally with LM Studio's new headless CLI and Claude Code

#36

Post by IceWreck »

Yukonv wrote: post #24
IceWreck wrote: post #15

Earlier quoted context omitted.

It does if you use an inference engine where you can offload some of the experts from VRAM to CPU RAM. That means I can fit a 35 billion param MoE in let's say 12 GB VRAM GPU + 16 gigs of memory.

With that you are taking a significant performance penalty and become severely I/O bottlenecked. I've been able to stream Qwen3.5-397B-A17B from my M5 Max (12 GB/s SSD Read) using the Flash MoE technique at the brisk pace of 10 tokens per second. As tokens are generated different experts need to be consulted resulting in a lot of I/O churn. So while feasible it's only great for batch jobs not interactive usage.

> So while feasible it's only great for batch jobs not interactive usage.

I mean yeah true but depends on how big the model is. The example I gave (Qwen 3.5 35BA3B) was fitting a 35B Q4 K_M (say 20 GB in size) model in 12 GB VRAM. With a 4070Ti + high speed 32 GB DDR5 ram you can easily get 700 token/sec prompt processing and 55-60 token/sec generation which is quite fast.

On the other hand if I try to fit a 120B model in 96 GB of DDR5 + the same 12 GB VRAM I get 2-5 token/sec generation.

Re: Running Gemma 4 locally with LM Studio's new headless CLI and Claude Code

#37

Post by superb_dev »

mbesto wrote: post #25
unsnap_biceps wrote: post #6

Earlier quoted context omitted.

lm studio offers an Anthropic compatible local endpoint, so you can point Claude code at it and it'll use your local model for it's requests, however, I've had a lot of problems with LM Studio and Claude code losing it's place. It'll think for awhile, come up with a plan, start to do it and then just halt in the middle. I'll ask it to continue and it'll do a small change and get stuck again. Using ollama's api doesn'…

I don't get why I would use Claude Code when OpenCode, Cursor, Zed, etc. all exist, are "free" and work with virtually any llm. Seems like a weird use case unless I'm missing something.

From my experience, Claude Code is just better. Although I recently started using Zed and it’s pretty good

Re: Running Gemma 4 locally with LM Studio's new headless CLI and Claude Code

#38

Post by zozbot234 »

IceWreck wrote: post #36
Yukonv wrote: post #24

Earlier quoted context omitted.

With that you are taking a significant performance penalty and become severely I/O bottlenecked. I've been able to stream Qwen3.5-397B-A17B from my M5 Max (12 GB/s SSD Read) using the Flash MoE technique at the brisk pace of 10 tokens per second. As tokens are generated different experts need to be consulted resulting in a lot of I/O churn. So while feasible it's only great for batch jobs not interactive usage.

> So while feasible it's only great for batch jobs not interactive usage. I mean yeah true but depends on how big the model is. The example I gave (Qwen 3.5 35BA3B) was fitting a 35B Q4 K_M (say 20 GB in size) model in 12 GB VRAM. With a 4070Ti + high speed 32 GB DDR5 ram you can easily get 700 token/sec prompt processing and 55-60 token/sec generation which is quite fast. On the other hand if I try to fit a 120B mod…

Your 120B model likely has way more active parameters, so it can probably only fit a few shared layers in the VRAM for your dGPU. You might be better off running that model on a unified memory platform, slower VRAM but a lot more of it.

Re: Running Gemma 4 locally with LM Studio's new headless CLI and Claude Code

#39

Post by pshirshov »

trvz wrote: post #26
pshirshov wrote: post #21

Earlier quoted context omitted.

For some reason, that doesn't work for me, claude never returns from some ill loop. Nemotron, glm and qwen 3.5 work just fine, gemma - doesn't.

Since that defaults to the q4 variant, try the q8 one: ollama launch claude --model gemma4:26b-a4b-it-q8_0

Even tried gemma4:31b and gemma4:31b with 128k context (I have 72GiB VRAM). Nothing. I'm cursed I guess. That's ollama-rocm if that matters (I had weird bugs on Vulkan, maybe gemma misbehaves on radeons somehow?..).

UPD: tried ollama-vulkan. It works, gemma4:31b-it-q8_0 with 64k context!

Re: Running Gemma 4 locally with LM Studio's new headless CLI and Claude Code

#40

Post by pshirshov »

asymmetric wrote: post #28

Is a framework desktop with >48GB of RAM a good machine to try this out?

Only for chat sessions, not for agentic coding. It's just too slow to be practical (10 minutes to answer a simple question about a 2k LoC project - and that's with a 5070 addon card).
Post reply on HN