21–29 of 29 posts
What hardware are you running the 30b model on? I guess it needs at least 24GB VRAM for decent inference speeds.
The general rule to follow is that you need as much VRAM as the model size. 30b models are usually around 19GB. So, most likely a GPU with 24GB of VRAM.
I was trying to get Claude code to work with llama.cpp but could never figure out anything functional. It always insisted on a phone home login for first time setup. In cline I’m getting better results with glm-4.7-flash than with qwen3-coder:30b
But with Qwen3-30B-A3B I get 20 tps in CC.
https://github.com/pchalasani/claude-code-tools/blob/main/do...
One tricky thing that took me a whole day to figure out is that using Claude Code in this setup was causing total network failures due to telemetry pings, so I had to set this env var to 1: CLAUDE_CODE_DISABLE_NONESSENTIAL_TRAFFIC
Earlier quoted context omitted.
What probably needs to exist is something like `llsed`. The invocation would be like this llsed --host 0.0.0.0 --port 8080 --map_file claude_to_openai.json --server https://openrouter.ai/api Where the json has something like { tag: ... from: ..., to: ..., params: ..., pre: ..., post: ...} So if one call is two, you can call multiple in the pre or post or rearrange things accordingly. This sounds like the proper separ…
Some unsolicited advice: Streaming support is tricky. I'd strip the streaming out when you proxy until everything else is solid.
This will be a bit challenging I'm sure but I agree, litellm and friends do too many things and take too long to get simple asks from
I've been pitching this suite I'm building as "GNU coreutils for the LLM era"
It's not sticking and nobody is hyped by it.
I don't know if I should keep going or if this is my same old pattern cropping up again of things I really really like but just kinda me
Earlier quoted context omitted.
The general rule to follow is that you need as much VRAM as the model size. 30b models are usually around 19GB. So, most likely a GPU with 24GB of VRAM.
But this also means tiny context windows. You can't fit gpt-oss:20b + more than a tiny file + instructions into 24GB
Earlier quoted context omitted.
Some unsolicited advice: Streaming support is tricky. I'd strip the streaming out when you proxy until everything else is solid.
Cool. Sounds good. Thanks. I'll do it. This will be a bit challenging I'm sure but I agree, litellm and friends do too many things and take too long to get simple asks from I've been pitching this suite I'm building as "GNU coreutils for the LLM era" It's not sticking and nobody is hyped by it. I don't know if I should keep going or if this is my same old pattern cropping up again of things I really really like but j…
The value comprehension market is small
So I'll need to surface it better or just do something else
The Anthropic API was already supported by llama.cpp (The project Ollama ripped off and typically lags in features by 3-6 months), which already works perfectly fine with Claude Code by setting a simple environment variable.
Earlier quoted context omitted.
But this also means tiny context windows. You can't fit gpt-oss:20b + more than a tiny file + instructions into 24GB
Gpt-oss is natively 4-bit, so you kinda can
What hardware are you running the 30b model on? I guess it needs at least 24GB VRAM for decent inference speeds.