Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Running Gemma 4 locally with LM Studio's new headless CLI and Claude Code

ai.georgeliu.com

Post reply on HN
Print view View original

111–120 of 121 posts

Re: Running Gemma 4 locally with LM Studio's new headless CLI and Claude Code

#111

Post by d4rkp4ttern »

selectodude wrote: post #93
peder wrote: post #86

Earlier quoted context omitted.

Did you have any Anthropic vs OpenAI specification issues with Claude Code? I have been using mlx_vlm and vMLX and I get 400 Bad Request errors from Claude Code. Presumably you're not seeing those issues with llama-server ?

I’ve jumped over to oMLX. A ton of rough edges but I think it’s the future.

At least for the Gemma4-26B-A4B, Token-gen speed with OMLX is far worse on my M1 Max 64GB Macbook, compared to llama-server:

  Quick benchmark on M1 Max 64GB, Gemma 4 26B-A4B (MoE), comparing matched dynamic 4-bit quants. Workload
  was Claude Code, which sends ~35K tokens of input context per request (system prompt + tools + user
  message):

  llama.cpp (unsloth/gemma-4-26B-A4B-it-GGUF:UD-Q4_K_XL, llama-server -fa on -c 131072 --jinja --temp 1.0
  --top-p 0.95 --top-k 64):
  - pp ≈ 395 tok/s
  - tg ≈ 40 tok/s

  oMLX (unsloth/gemma-4-26b-a4b-it-UD-MLX-4bit, omlx serve --model-dir ~/models/omlx, with
  sampling.max_context_window and max_tokens bumped to 131072 in ~/.omlx/settings.json):
  - pp ≈ 350 tok/s
  - tg ≈ 5–13 tok/s

  Same model family and quant tier. Prompt processing is comparable, but oMLX's token generation is 3–7x
  slower than llama.cpp's Metal backend. Counter-intuitive given MLX is Apple's native ML framework.

Re: Running Gemma 4 locally with LM Studio's new headless CLI and Claude Code

#112

Post by d4rkp4ttern »

peder wrote: post #109
selectodude wrote: post #93

Earlier quoted context omitted.

I’ve jumped over to oMLX. A ton of rough edges but I think it’s the future.

Have you been using `omlx serve`? If so, how are you bumping up the max context size? I'm not seeing a param to go above 32k?

you can set it in the .omlx/settings.json - ask a code-agent to figure it out by pointing it at the omlx repo

Re: Running Gemma 4 locally with LM Studio's new headless CLI and Claude Code

#113

Post by pseudosavant »

I want local models to succeed, but today the gap vs cloud models still seems continually too large. Even with a $2k GPU or a $4k MBP, the quality and speed tradeoff usually isn’t sensible.

Credit to Google for releasing Gemma 4, though. I’d love to see local models reach the point where a 32 GB machine can handle high quality agentic coding at a practical speed.

Re: Running Gemma 4 locally with LM Studio's new headless CLI and Claude Code

#114

Post by bckr »

vkou wrote: post #106
bckr wrote: post #96

Earlier quoted context omitted.

I think it’s ALL getting commoditized. The winners here are engineers (who are onboard with the agentic surge) and, hopefully, users who get more and better software.

> hopefully, users who get more and better software. Users are definitely going to get more software and more features and redesigns in the software they use, but I have strong doubts that it's going to get better. If pre-LLM developer productivity was used to build all sorts of deranged anti-user promo-padding bullshit, imagine how much more of it we can do with a 2x more productive employee base.

yes. But also some stuff will be amazing.

Re: Running Gemma 4 locally with LM Studio's new headless CLI and Claude Code

#115

Post by ffsm8 »

pseudosavant wrote: post #113

I want local models to succeed, but today the gap vs cloud models still seems continually too large. Even with a $2k GPU or a $4k MBP, the quality and speed tradeoff usually isn’t sensible. Credit to Google for releasing Gemma 4, though. I’d love to see local models reach the point where a 32 GB machine can handle high quality agentic coding at a practical speed.

Fwiw, the real reason we don't have 100+ GB GPUs is because Nvidia likes to segment their markets. They could sell the consumer cards with 200gb gddr RAM on it, they just know that'd eat into their enterprise offering which is quiet literally all their profit margin (which I may add is gargantuan as of 2025)

Re: Running Gemma 4 locally with LM Studio's new headless CLI and Claude Code

#116

Post by alfiedotwtf »

PSA: For those getting stuck in a repetitive loop or just stopping without completing a task, try the interactive template. I just tried it now and it's blowing my already impressive results out of the water (llama.cpp):

    --jinja --chat-template-file models/templates/google-gemma-4-31B-it-interleaved.jinja

Re: Running Gemma 4 locally with LM Studio's new headless CLI and Claude Code

#117

Post by rachel_rig »

unsnap_biceps wrote: post #6
jonplackett wrote: post #5

So wait what is the interaction between Gemma and Claude?

lm studio offers an Anthropic compatible local endpoint, so you can point Claude code at it and it'll use your local model for it's requests, however, I've had a lot of problems with LM Studio and Claude code losing it's place. It'll think for awhile, come up with a plan, start to do it and then just halt in the middle. I'll ask it to continue and it'll do a small change and get stuck again. Using ollama's api doesn'…

[flagged]

Re: Running Gemma 4 locally with LM Studio's new headless CLI and Claude Code

#118

Post by bicepjai »

Totally agree lmstudio headless server on a remote machine but control models from your laptop is an amazing workflow. But Gemma 4 was not a good model atleast in my trials “find me the largest text file in all of the current sub folders” it went on a loopy tool call for ever even with Q8

Re: Running Gemma 4 locally with LM Studio's new headless CLI and Claude Code

#119

Post by unstatusthequo »

selectodude wrote: post #93
peder wrote: post #86

Earlier quoted context omitted.

Did you have any Anthropic vs OpenAI specification issues with Claude Code? I have been using mlx_vlm and vMLX and I get 400 Bad Request errors from Claude Code. Presumably you're not seeing those issues with llama-server ?

I’ve jumped over to oMLX. A ton of rough edges but I think it’s the future.

Check out vMLX if you use Apple Silicon. https://github.com/jjang-ai/mlxstudio
Post reply on HN