I could see a future in which the major AI labs run a local LLM to offload much of the computational effort currently undertaken in the cloud, leaving the heavy lifting to cloud-hosted models and the easier stuff for local inference.
Running Gemma 4 locally with LM Studio's new headless CLI and Claude Code
91–100 of 121 posts
Re: Running Gemma 4 locally with LM Studio's new headless CLI and Claude Code
#92I could see a future in which the major AI labs run a local LLM to offload much of the computational effort currently undertaken in the cloud, leaving the heavy lifting to cloud-hosted models and the easier stuff for local inference.
wouldnt that be counter to their whole business model?
Re: Running Gemma 4 locally with LM Studio's new headless CLI and Claude Code
#93You can use llama.cpp server directly to serve local LLMs and use them in Claude Code or other CLI agents. I’ve collected full setup instructions for Gemma4 and other recent open-weight LLMs here, tested on my M1 Max 64 GB MacBook: https://pchalasani.github.io/claude-code-tools/integrations/... The 26BA4B is the most interesting to run on such hardware, and I get nearly double the token-gen speed (40 tok/s) compared…
Did you have any Anthropic vs OpenAI specification issues with Claude Code? I have been using mlx_vlm and vMLX and I get 400 Bad Request errors from Claude Code. Presumably you're not seeing those issues with llama-server ?
I’ve jumped over to oMLX. A ton of rough edges but I think it’s the future.
Re: Running Gemma 4 locally with LM Studio's new headless CLI and Claude Code
#94[flagged]
Sounds like the exact opposite, models are being commoditized while the harness and tooling around a model is what actually gets significant gains, especially with RL around specific models.
For example, this article was posted recently, Improving 15 LLMs at Coding in One Afternoon. Only the Harness Changed [0].
Re: Running Gemma 4 locally with LM Studio's new headless CLI and Claude Code
#95Running Gemma 4 with llama.cpp and Swival:
$ llama-server --reasoning auto --fit on -hf unsloth/gemma-4-26B-A4B-it-GGUF:UD-Q4_K_XL --temp 1.0 --top-p 0.95 --top-k 64
$ uvx swival --provider llamacpp
Done.
Re: Running Gemma 4 locally with LM Studio's new headless CLI and Claude Code
#96[flagged]
Sounds like the exact opposite, models are being commoditized while the harness and tooling around a model is what actually gets significant gains, especially with RL around specific models. For example, this article was posted recently, Improving 15 LLMs at Coding in One Afternoon. Only the Harness Changed [0]. [0] https://news.ycombinator.com/item?id=46988596
I think it’s ALL getting commoditized. The winners here are engineers (who are onboard with the agentic surge) and, hopefully, users who get more and better software.
Re: Running Gemma 4 locally with LM Studio's new headless CLI and Claude Code
#97Seems like this might be a great way to do web software testing. We’ve had Selenium and Puppeteer for a long time but they are a bit brittle with respect to the web design. Change something about the design and there’s a high likelihood that a test will break. Seems like this might be able to be smarter about adapting to changes. That’s also a great use for a smaller model like this.
Re: Running Gemma 4 locally with LM Studio's new headless CLI and Claude Code
#98You can use llama.cpp server directly to serve local LLMs and use them in Claude Code or other CLI agents. I’ve collected full setup instructions for Gemma4 and other recent open-weight LLMs here, tested on my M1 Max 64 GB MacBook: https://pchalasani.github.io/claude-code-tools/integrations/... The 26BA4B is the most interesting to run on such hardware, and I get nearly double the token-gen speed (40 tok/s) compared…
Did you have any Anthropic vs OpenAI specification issues with Claude Code? I have been using mlx_vlm and vMLX and I get 400 Bad Request errors from Claude Code. Presumably you're not seeing those issues with llama-server ?
Correct, no issues because since at least a few months, llama.cpp/server exposes an Anthropic messages API at v1/messages, in addition to the OpenAI-compatible API at v1/chat/completions. Claude Code uses the former.
Re: Running Gemma 4 locally with LM Studio's new headless CLI and Claude Code
#99[flagged]
Sounds like the exact opposite, models are being commoditized while the harness and tooling around a model is what actually gets significant gains, especially with RL around specific models. For example, this article was posted recently, Improving 15 LLMs at Coding in One Afternoon. Only the Harness Changed [0]. [0] https://news.ycombinator.com/item?id=46988596
[dead]
Re: Running Gemma 4 locally with LM Studio's new headless CLI and Claude Code
#100[flagged]
I find MCP beneficial too, but do be aware of token usage. With a naive implementation MCP can use significantly more input tokens (and context) than equivalent skills would. With a handful of third party MCPs I’ve seen tens of thousands of tokens used before I’ve started anything. Here’s an article from Anthropic explaining why, but it is 5 months old so perhaps it's irrelevant ancient history at this point. https:/…
[dead]