Just FYI, MoE doesn't really save (V)RAM. You still need all weights loaded in memory, it just means you consult less per forward pass. So it improves tok/s but not vram usage.
Running Gemma 4 locally with LM Studio's new headless CLI and Claude Code
11–20 of 121 posts
Re: Running Gemma 4 locally with LM Studio's new headless CLI and Claude Code
#12Earlier quoted context omitted.
lm studio offers an Anthropic compatible local endpoint, so you can point Claude code at it and it'll use your local model for it's requests, however, I've had a lot of problems with LM Studio and Claude code losing it's place. It'll think for awhile, come up with a plan, start to do it and then just halt in the middle. I'll ask it to continue and it'll do a small change and get stuck again. Using ollama's api doesn'…
Claude Code is fairly notoriously token inefficient as far as coding agent/harnesses go (i come from aider pre-CC). It's only viable because the Max subscriptions give you approximately unlimited token budget, which resets in a few hours even if you hit the limit. But this also only works because cloud models have massive token windows (1M tokens on opus right now) which is a bit difficult to make happen locally with…
Can't you use Claude caveman mode?
Re: Running Gemma 4 locally with LM Studio's new headless CLI and Claude Code
#13Using Claude Code seems like a popular frontend currently, I wonder how long until Anthropic releases an update to make it a little to a lot less turn-key? They've been very clear that they aren't exactly champions of this stuff being used outside of very specific ways.
Right now it suits them down to the ground. You pay for the product and you don’t cost their servers anything.
You don't pay anything to use Claude Code as a front end to non-Anthropic models
Re: Running Gemma 4 locally with LM Studio's new headless CLI and Claude Code
#14Earlier quoted context omitted.
lm studio offers an Anthropic compatible local endpoint, so you can point Claude code at it and it'll use your local model for it's requests, however, I've had a lot of problems with LM Studio and Claude code losing it's place. It'll think for awhile, come up with a plan, start to do it and then just halt in the middle. I'll ask it to continue and it'll do a small change and get stuck again. Using ollama's api doesn'…
Claude Code is fairly notoriously token inefficient as far as coding agent/harnesses go (i come from aider pre-CC). It's only viable because the Max subscriptions give you approximately unlimited token budget, which resets in a few hours even if you hit the limit. But this also only works because cloud models have massive token windows (1M tokens on opus right now) which is a bit difficult to make happen locally with…
I use CC at work, so I haven't explored other options. Is there a better one to use locally? I presumed they were all going to be pretty similar.
Re: Running Gemma 4 locally with LM Studio's new headless CLI and Claude Code
#15Just FYI, MoE doesn't really save (V)RAM. You still need all weights loaded in memory, it just means you consult less per forward pass. So it improves tok/s but not vram usage.
It does if you use an inference engine where you can offload some of the experts from VRAM to CPU RAM.
That means I can fit a 35 billion param MoE in let's say 12 GB VRAM GPU + 16 gigs of memory.
Re: Running Gemma 4 locally with LM Studio's new headless CLI and Claude Code
#16Using Claude Code seems like a popular frontend currently, I wonder how long until Anthropic releases an update to make it a little to a lot less turn-key? They've been very clear that they aren't exactly champions of this stuff being used outside of very specific ways.
Is it not about the same as using OpenCode?
And is running a local model with Claude Code actually usable for any practical work compared to the hosted Anthropic models?
Re: Running Gemma 4 locally with LM Studio's new headless CLI and Claude Code
#17Re: Running Gemma 4 locally with LM Studio's new headless CLI and Claude Code
#18Using Claude Code seems like a popular frontend currently, I wonder how long until Anthropic releases an update to make it a little to a lot less turn-key? They've been very clear that they aren't exactly champions of this stuff being used outside of very specific ways.
I think CC is popular because they are catering to the common denominator programmer and are going to continue to do that, not because CC is particularly turn-key.
Re: Running Gemma 4 locally with LM Studio's new headless CLI and Claude Code
#19Using Claude Code seems like a popular frontend currently, I wonder how long until Anthropic releases an update to make it a little to a lot less turn-key? They've been very clear that they aren't exactly champions of this stuff being used outside of very specific ways.
I don't think there is any incentive to do so right now because the open models aren't as good. The vast majority of businesses are going to just pay the extra cost for access to a frontier model. The model is what gives them a competitive advantage, not the harness. The harness is a lot easier to replicate than Opus.
There are benefits too. Some developers might learn to use Claude Code outside of work with cheaper models and then advocate for using Claude Code at work (where their companies will just buy access from Anthropic, Bedrock, etc). Similar to how free ESXi licenses for personal use helped infrastructure folks gain skills with that product which created a healthy supply of labor and VMware evangelists that were eager to spread the gospel. Anthropic can't just give away access to Claude models because of cost so there is use in allowing alternative ways for developers to learn how to use Claude Code and develop a workflow with it.
Re: Running Gemma 4 locally with LM Studio's new headless CLI and Claude Code
#20Just FYI, MoE doesn't really save (V)RAM. You still need all weights loaded in memory, it just means you consult less per forward pass. So it improves tok/s but not vram usage.
You never need to have all weights in memory. You can swap them in from RAM, disk, the network, etc. MOE reduces the amount of data that will need to be swapped in for the next forward pass.