[flagged]
You could always point Claude Code and open code at a local http endpoint
Running Gemma 4 locally with LM Studio's new headless CLI and Claude Code
101–110 of 121 posts
Re: Running Gemma 4 locally with LM Studio's new headless CLI and Claude Code
#102I could see a future in which the major AI labs run a local LLM to offload much of the computational effort currently undertaken in the cloud, leaving the heavy lifting to cloud-hosted models and the easier stuff for local inference.
wouldnt that be counter to their whole business model?
I don't think so. Acquiring hardware for inference is a chokepoint on growth. If they can offload some inference to the customer's machine, that allows them to use more of their online capacity to generate money.
Re: Running Gemma 4 locally with LM Studio's new headless CLI and Claude Code
#103Just FYI, MoE doesn't really save (V)RAM. You still need all weights loaded in memory, it just means you consult less per forward pass. So it improves tok/s but not vram usage.
This confused me at first as well.. inactive experts skip compute, but weights are sill loaded. So memory does not shrink at all.
I found this visualisation helpful - https://vectree.io/c/sparse-activation-patterns-and-memory-e...
Re: Running Gemma 4 locally with LM Studio's new headless CLI and Claude Code
#104Earlier quoted context omitted.
Did you have any Anthropic vs OpenAI specification issues with Claude Code? I have been using mlx_vlm and vMLX and I get 400 Bad Request errors from Claude Code. Presumably you're not seeing those issues with llama-server ?
I’ve jumped over to oMLX. A ton of rough edges but I think it’s the future.
Same. Opencode + oMLX (0.3.4) + unsloth-Qwen3-Coder-Next-mlx-8bit on my M5 Max w 128GB is the sweet spot for me locally. The prompt decode caching keeps things coherent and fast even when contexts get north of 100k tokens.
Re: Running Gemma 4 locally with LM Studio's new headless CLI and Claude Code
#105Local models are finally starting to feel pleasant instead of just "possible." The headless LM Studio flow is especially nice because it makes local inference usable from real tools instead of as a demo. Related note from someone building in this space: I've been working on cloclo ( https://www.npmjs.com/package/cloclo ), an open-source coding agent CLI, and this is exactly the direction I'm excited about. It nativel…
How does cloclo differ from pi-mono?
pi-mono is a great toolkit — coding agent CLI, unified LLM API, web UI, Slack bot, vLLM pods.
cloclo is a runtime for agent toolkits. You plug it into your own agents and it gives them multi-agent orchestration (AICL protocol), 13 providers, skill registry, native browser/docs/phone tools, memory, and an NDJSON bridge. Zero native deps.
Re: Running Gemma 4 locally with LM Studio's new headless CLI and Claude Code
#106Earlier quoted context omitted.
Sounds like the exact opposite, models are being commoditized while the harness and tooling around a model is what actually gets significant gains, especially with RL around specific models. For example, this article was posted recently, Improving 15 LLMs at Coding in One Afternoon. Only the Harness Changed [0]. [0] https://news.ycombinator.com/item?id=46988596
I think it’s ALL getting commoditized. The winners here are engineers (who are onboard with the agentic surge) and, hopefully, users who get more and better software.
> hopefully, users who get more and better software.
Users are definitely going to get more software and more features and redesigns in the software they use, but I have strong doubts that it's going to get better.
If pre-LLM developer productivity was used to build all sorts of deranged anti-user promo-padding bullshit, imagine how much more of it we can do with a 2x more productive employee base.
Re: Running Gemma 4 locally with LM Studio's new headless CLI and Claude Code
#107Seems like this might be a great way to do web software testing. We’ve had Selenium and Puppeteer for a long time but they are a bit brittle with respect to the web design. Change something about the design and there’s a high likelihood that a test will break. Seems like this might be able to be smarter about adapting to changes. That’s also a great use for a smaller model like this.
Yeah. I think that's an interesting use case. Especially if I can kick it off or schedule it when I'm not actively working. Inference speed (especially with tool calling involved) won't be great on my machines, but if I schedule nightly usability tests of dev sites while I sleep, that could be really cool.
Re: Running Gemma 4 locally with LM Studio's new headless CLI and Claude Code
#108Seems like this might be a great way to do web software testing. We’ve had Selenium and Puppeteer for a long time but they are a bit brittle with respect to the web design. Change something about the design and there’s a high likelihood that a test will break. Seems like this might be able to be smarter about adapting to changes. That’s also a great use for a smaller model like this.
Yeah. I think that's an interesting use case. Especially if I can kick it off or schedule it when I'm not actively working. Inference speed (especially with tool calling involved) won't be great on my machines, but if I schedule nightly usability tests of dev sites while I sleep, that could be really cool.
You’re right about inference speed being a concern. I was assuming it’s a small model but even then, one of the browser automation frameworks is going to be faster.
Re: Running Gemma 4 locally with LM Studio's new headless CLI and Claude Code
#109Earlier quoted context omitted.
Did you have any Anthropic vs OpenAI specification issues with Claude Code? I have been using mlx_vlm and vMLX and I get 400 Bad Request errors from Claude Code. Presumably you're not seeing those issues with llama-server ?
I’ve jumped over to oMLX. A ton of rough edges but I think it’s the future.
Have you been using `omlx serve`? If so, how are you bumping up the max context size? I'm not seeing a param to go above 32k?
Re: Running Gemma 4 locally with LM Studio's new headless CLI and Claude Code
#110Earlier quoted context omitted.
> So while feasible it's only great for batch jobs not interactive usage. I mean yeah true but depends on how big the model is. The example I gave (Qwen 3.5 35BA3B) was fitting a 35B Q4 K_M (say 20 GB in size) model in 12 GB VRAM. With a 4070Ti + high speed 32 GB DDR5 ram you can easily get 700 token/sec prompt processing and 55-60 token/sec generation which is quite fast. On the other hand if I try to fit a 120B mod…
Your 120B model likely has way more active parameters, so it can probably only fit a few shared layers in the VRAM for your dGPU. You might be better off running that model on a unified memory platform, slower VRAM but a lot more of it.
Yep, I understand I was giving an example to the person I was replying to.