551–560 of 620 posts
Earlier quoted context omitted.
Not OP, but I am seeing up to 260 tokens/second output at c=1 with the recipe at https://github.com/local-inference-lab/rtx6kpro/blob/master/... using 4x 6k cards. Average is more like 200. There may be a way to get the 2-bit quantized version running even faster on a pair of them.
Thank you. Useful to know. Clipped on top by reduce, I assume.
Earlier quoted context omitted.
In my personal experience, I wouldn't bother with 16GB cards for coding -- the useful models are _slightly_ too large to work at any reasonable speed
That's not my experience, and the trajectory is good anyway - what doesn't work perfectly today will be just fine in a few months. In a quickly moving field, it's amazing how much money one can save by overcoming FOMO and not living on the bleeding edge. It's like waiting for Steam sales, the games will be just as good.
Earlier quoted context omitted.
So, one of the ways that this problem manifests is that most local models aren't trained on preserving the full reasoning between turns. Every turn, they skip passing the reasoning trace from previous turns to the the LLM. So if on one turn you have a long interleaved chain of reasoning and tool calls, then it responds to you, and then you give a new prompt to fix something, it has to re-process all of those tools ca…
thats a harness issue not a model issue. eg i have my own reasoninf harness that forced persisted cot
The jinja template is what renders the openai-format request sent by the harness, into the actual string of text that will be tokenized and fed to the model. For models without preserve thinking support, the jinja template drops the reasoning from all but the current turn.
Here is the default jinja for Gemma 4: https://huggingface.co/google/gemma-4-31B-it/blob/main/chat_...
{#- Render reasoning/reasoning_content as thinking channel -#}
{%- set thinking_text = message.get('reasoning') or message.get('reasoning_content') -%}
{%- if thinking_text and loop.index0 > ns_turn.last_user_idx and message.get('tool_calls') -%}
{{- 'thought\n' + thinking_text + '\n' -}}
{%- endif -%}
Here's Qwen 3.6 by comparison: https://huggingface.co/Qwen/Qwen3.6-35B-A3B/blob/main/chat_t...
{%- if (preserve_thinking is defined and preserve_thinking is true) or (loop.index0 > ns.last_query_index) %}
{{- '' + message.role + '\n\n' + reasoning_content + '\n\n\n' + content }}
{%- else %}
{{- '' + message.role + '\n' + content }}
{%- endif %}
It's possible to modify the jinja file that you're using with a model. Some people do that with models that haven't been specifically trained for it, and report good results; but some report that because it wasn't trained for that, they get worse results if they include thinking from previous turns.
So for models like Gemma, you would have to modify the default jinja to enable this. For Qwen, you can just set the preserve_thinking flag to get this behavior; and apparently they have trained in this mode so you get better results than models that have not trained this way.
Earlier quoted context omitted.
> unsloth/Qwen3.6-35B-A3B-MTP-GGUF I've actually tried this exact same model locally as well.. albeit on just a single 3090 at 128k context and I got around 40-60tok/s with Q4_K quantization. The thing that bugged me the most was really the quality of the output on moderately complex real-world coding tasks. Having to switch between "prompt/vibe" and "manually implement" is such a big context switch burden, because y…
aider sucks tbh... you should invest time in learning how to customise pi. every other harness is crap and hype.
Earlier quoted context omitted.
> unsloth/Qwen3.6-35B-A3B-MTP-GGUF I've actually tried this exact same model locally as well.. albeit on just a single 3090 at 128k context and I got around 40-60tok/s with Q4_K quantization. The thing that bugged me the most was really the quality of the output on moderately complex real-world coding tasks. Having to switch between "prompt/vibe" and "manually implement" is such a big context switch burden, because y…
See, that makes it sound better in my world: I'm doing all 'manually implement' all the time, and have no interest in becoming the ten millionth manager to hit the software dev landscape. It boggles my mind that people think this is a win. I regularly use a pocket calculator: either a physical one, or Apple's Calculator app if I want more decimal places. 'Too stupid' isn't a thing for me if it can cough up some math…
Is it _objectively_ more productive? I doubt there's a clear-cut answer in the long run (my main suspicion is that since you're essentially creating 10x unnecessary complexity, you'll likely never recover from all the cruft and maintenance kills you in the end - maybe people will find solutions for that though).
Earlier quoted context omitted.
That's not my experience, and the trajectory is good anyway - what doesn't work perfectly today will be just fine in a few months. In a quickly moving field, it's amazing how much money one can save by overcoming FOMO and not living on the bleeding edge. It's like waiting for Steam sales, the games will be just as good.
Curious what model you're using that works well on a 16GB card? I very much want to use my 5080 for inference, but everything I've tried so far has either just not been good enough or painfully slow.
I have a 5080 too! For me, the key has been dropping Ollama for Llama.cpp, which is not particularly scary to configure anymore and just skyrocketed performance. I download the models with LM Studio, then run them with llama.cpp.
the harnesses themselves are just as important as the models...different harnesses give different responses with the same prompt, same model...
if you have the 20/mnth claude sub or codex, you really should be using that to build a good local harness for yourself... claude won't be 20$ forever
build the stack first! when you get that new comp with massive ram, youre already set, just run a larger model!
big cloud models are incredibly good at building and teaching about local ai!
have fun in the rabbit hole!
if you are memory constrained like me, check out my custom models https://huggingface.co/macwhisperer
Not “local” and not interactive coding but sharing since it might be helpful. I have 2x RTX Pro 6000 Blackwell running DeepSeek V4 Flash. I get 160 tok/s raw but it’s a reasoning model. For my use case, I have it auto-write code and another system auto-review the code. I occasionally use it with pi to write some code and it’s blazing fast but it’s mostly habit that keeps me with CC and Codex.
What quant?
Earlier quoted context omitted.
2x RTX3090 are around $4400. Without any electricity costs or other parts, that's 3.6 years of $100/m claude.
You can get 60tps with three 1080tis and the sparse model, and I bet two 16gb 5060tis would do the same for ~1200. One 3090 is enough for a useful system, even on an old am4 host.
Earlier quoted context omitted.
Given your knowledge on this - do you think we'll see an open source model with Opus levels of capability? IMO if/when this happens - I would 100% stop using Anthropic.
If you believe the benchmarks, Qwen 3.6 35B-A3B already outperforms Claude 4 Opus. Now, there's a bit of a degree to which some of the open source models do some benchmaxxing, and bigger models with more params may always feel like they have more depth. But anyhow, right now you have something that is arguably comparable to Claude 4 Opus on your laptop. I can't really compare myself because I never used it. It looks…