Why your local LLM feels dumber than it is

Post by aktenlage » Fri, Aug 28, 2026, 7:31 PM UTC

tharkun__ wrote: post #206

tharkun__ wrote: post #197

Earlier quoted context omitted.

If I use the 12B Unified (dense) model I mentioned without MTP, then I get 37t/s, input ~700t/s. It's all still quite frustrating in the end, like a Claude from a very long time ago by now but usable. If I want 64k context, I can't use MTP. I still haven't decided whether I'd rather have 37t/s but it's "less dumb" or I want MTP speed but it's going off the rails more. All of this is also with `-ctv q4_0 -ctk q4_0)`,…

Haha, OK, so it processes input tokens at 55t/s if started like the 12B. That's so slow that I don't even have the patience to wait for the context of the session I aborted mid-processing to test this, to load. Heck loading the system prompt alone is not fun. That said, after that it seems to run at the same ~35t/s speed but , I was able to start it up with a 128k context (and no KV quantization), humm humm. I am now…

Post by tharkun__ » Tue, Sep 01, 2026, 3:36 AM UTC

aktenlage wrote: post #231

tharkun__ wrote: post #206

Earlier quoted context omitted.

Haha, OK, so it processes input tokens at 55t/s if started like the 12B. That's so slow that I don't even have the patience to wait for the context of the session I aborted mid-processing to test this, to load. Heck loading the system prompt alone is not fun. That said, after that it seems to run at the same ~35t/s speed but , I was able to start it up with a 128k context (and no KV quantization), humm humm. I am now…

Thank you, very interesting to read. I haven't considered batch sizes at all up to now. Seems I should look into it. How did you get the prediction that good? The default spec-draft-max-n is 3 and if I go beyond, the output rate stays the same or gets lower any magic spell you could share?