Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Running Gemma 4 locally with LM Studio's new headless CLI and Claude Code

ai.georgeliu.com

Post reply on HN
Print view View original

61–70 of 121 posts

Re: Running Gemma 4 locally with LM Studio's new headless CLI and Claude Code

#61

Post by panagathon »

mbesto wrote: post #25
unsnap_biceps wrote: post #6

Earlier quoted context omitted.

lm studio offers an Anthropic compatible local endpoint, so you can point Claude code at it and it'll use your local model for it's requests, however, I've had a lot of problems with LM Studio and Claude code losing it's place. It'll think for awhile, come up with a plan, start to do it and then just halt in the middle. I'll ask it to continue and it'll do a small change and get stuck again. Using ollama's api doesn'…

I don't get why I would use Claude Code when OpenCode, Cursor, Zed, etc. all exist, are "free" and work with virtually any llm. Seems like a weird use case unless I'm missing something.

> I don't get why I would use Claude Code when OpenCode, Cursor, Zed, etc. all exist, are "free" and work with virtually any llm. Seems like a weird use case unless I'm missing something.

I'm with you on this. I've tried Gemma and Claude code and it's not good. Forgets it can use bash!

However, Gemma running locally with Pi as the harness is a beast.

Re: Running Gemma 4 locally with LM Studio's new headless CLI and Claude Code

#62

Post by nl »

pshirshov wrote: post #40
asymmetric wrote: post #28

Is a framework desktop with >48GB of RAM a good machine to try this out?

Only for chat sessions, not for agentic coding. It's just too slow to be practical (10 minutes to answer a simple question about a 2k LoC project - and that's with a 5070 addon card).

Doesn't the framework desktop have a Ryzen 395 AI? That's a unified memory architecture like the Macs.

Re: Running Gemma 4 locally with LM Studio's new headless CLI and Claude Code

#63

Post by tatrions »

keerthiko wrote: post #9
unsnap_biceps wrote: post #6

Earlier quoted context omitted.

lm studio offers an Anthropic compatible local endpoint, so you can point Claude code at it and it'll use your local model for it's requests, however, I've had a lot of problems with LM Studio and Claude code losing it's place. It'll think for awhile, come up with a plan, start to do it and then just halt in the middle. I'll ask it to continue and it'll do a small change and get stuck again. Using ollama's api doesn'…

Claude Code is fairly notoriously token inefficient as far as coding agent/harnesses go (i come from aider pre-CC). It's only viable because the Max subscriptions give you approximately unlimited token budget, which resets in a few hours even if you hit the limit. But this also only works because cloud models have massive token windows (1M tokens on opus right now) which is a bit difficult to make happen locally with…

[flagged]

Re: Running Gemma 4 locally with LM Studio's new headless CLI and Claude Code

#65

Post by ac29 »

pshirshov wrote: post #40
asymmetric wrote: post #28

Is a framework desktop with >48GB of RAM a good machine to try this out?

Only for chat sessions, not for agentic coding. It's just too slow to be practical (10 minutes to answer a simple question about a 2k LoC project - and that's with a 5070 addon card).

This article is about a MoE model with only 4B active parameters, it shouldn't take 10 minutes to answer a question about a small project.

I measured a 4bit quant of this model at 1300t/s prefill and ~60t/s decode on Ryzen 395+.

Re: Running Gemma 4 locally with LM Studio's new headless CLI and Claude Code

#66

Post by andhuman »

unsnap_biceps wrote: post #14
keerthiko wrote: post #9

Earlier quoted context omitted.

Claude Code is fairly notoriously token inefficient as far as coding agent/harnesses go (i come from aider pre-CC). It's only viable because the Max subscriptions give you approximately unlimited token budget, which resets in a few hours even if you hit the limit. But this also only works because cloud models have massive token windows (1M tokens on opus right now) which is a bit difficult to make happen locally with…

I use CC at work, so I haven't explored other options. Is there a better one to use locally? I presumed they were all going to be pretty similar.

I have good experience with Mistral Vibe.

Re: Running Gemma 4 locally with LM Studio's new headless CLI and Claude Code

#67

Post by seifbenayed1992 »

Local models are finally starting to feel pleasant instead of just "possible." The headless LM Studio flow is especially nice because it makes local inference usable from real tools instead of as a demo.

Related note from someone building in this space: I've been working on cloclo (https://www.npmjs.com/package/cloclo), an open-source coding agent CLI, and this is exactly the direction I'm excited about. It natively supports LM Studio, Ollama, vLLM, Jan, and llama.cpp as providers alongside cloud models, so you can swap between local and hosted backends without changing how you work.

Feels like we're getting closer to a good default setup where local models are private/cheap enough to use daily, and cloud models are still there when you need the extra capability.

Re: Running Gemma 4 locally with LM Studio's new headless CLI and Claude Code

#68

Post by alfiedotwtf »

pshirshov wrote: post #39
trvz wrote: post #26

Earlier quoted context omitted.

Since that defaults to the q4 variant, try the q8 one: ollama launch claude --model gemma4:26b-a4b-it-q8_0

Even tried gemma4:31b and gemma4:31b with 128k context (I have 72GiB VRAM). Nothing. I'm cursed I guess. That's ollama-rocm if that matters (I had weird bugs on Vulkan, maybe gemma misbehaves on radeons somehow?..). UPD: tried ollama-vulkan. It works, gemma4:31b-it-q8_0 with 64k context!

The default context is 128k for the smaller Gemma 4’s and 256k for the bigger ones, so you’re cutting off context and it doesn’t know how to continue.

Bump it to native (or -c 0 may work too)

Re: Running Gemma 4 locally with LM Studio's new headless CLI and Claude Code

#70

Post by alfiedotwtf »

Someone1234 wrote: post #7

Using Claude Code seems like a popular frontend currently, I wonder how long until Anthropic releases an update to make it a little to a lot less turn-key? They've been very clear that they aren't exactly champions of this stuff being used outside of very specific ways.

Yet Codex specifically aims out to be compatible with all backends! Up until Gemma 4 though it’s been pretty solid, but totally fails with unknown tool (I’m guessing a template issue)
Post reply on HN