Why your local LLM feels dumber than it is

Maybe some were fixed, but:

1) Shipping with 2k default context window for the longest time, w/o any warning and being not easy to change (like any other setting). Totally made a lot of people think local LLMs are dumb as rocks. Just checked, still not fixed -- defaults to 4K if less than 24GB VRAM: https://docs.ollama.com/context-length

2) Registry mistrust due to Deepseek R1 naming. What model do you download/start with `ollama run deepseek-r1`? Not Deepseek R1, but this "for research purposes" thinking finetune of Llama 3 released alongside R1 paper: https://huggingface.co/deepseek-ai/DeepSeek-R1-Distill-Llama... . For actual R1 you have to pull `deepseek-r1:671b`

3) Can't reuse existing ggufs you have. If you point it to gguf, it would create a copy of it in ollama folder.

4) Doesn't accept engine parameters from CLI args. Only env variables or modelfile. But some things are only in env variables (e.g. KV cache quantization). And even then not even close to what's available in llama.cpp.

5) Often meaningfully slower than llama.cpp

6) VC-funded wrapper for llama.cpp. For long time had questionable attribution to it. Now looks good.

> (and also because VLLM is not super Windows friendly)

llama.cpp is Windows friendly and fairly easy. Not GUI app with installer, but easy. Zip with binaries are on github, run

  llama-server.exe -hf unsloth/gemma-4-12B-it-qat-GGUF:UD-Q4_K_XL

will pull the model you expect from hugging face (or -m for manually downloaded .gguf) and start web server with API and HTML chat app.

There's quite a few tangential features that must be implemented correctly or risk affecting the LLM output in significant ways.

Parsing/encoding is one example: A couple of months ago I've debugged a reasoning loop bug in Step 3.7 Flash on llama.cpp that was caused by the parser capturing an extra `\n` as part of a reasoning block. It was something that only manifested at longer multi-turn agentic sessions, and the extra linefeed was steering the model into making reasoning self corrections that only got worse with longer sessions (more details about this issue: https://github.com/ggml-org/llama.cpp/issues/24181#issuecomm...)

No inference engine is perfect, but I feel that llama.cpp is the most reliable way to run language models locally.

I believe whoever lays down the best structure around LLM will take the lead. Proven in Anthropic vs OpenAI.

That's funny, because I just went through the opposite.

I got llama.cpp working with qwen3.6 and qwen3.8 by Googling and manually adjusting things according to reddit posts and Google not-really-helpful AI suggestions.

I tried settings up per-model stuff in settings.json, but again Google got in my way, and llama.cpp having 2 different settings.json (and Google lying about where 1 goes) made it far too difficulty to figure out. I spent hours on it.

Then I got fed up and asked Claude.

Immediately, it told me that the winget version of llama.cpp is for Vulkan, and I needed a different one and pointed at it. It doubled my speed.

Then it figured out what I was doing wrong with settings.json (wrong spot, global settings can't go in the per-model file, etc etc) and fixed all that, and got it working.

Then it tuned it somewhat.

Then I showed it the official settings pages for both models, and it undid the tuning and all the damage I had done with my tinkering, and got everything working.

In 30 minutes.

It was absolutely amazing.

Every time I see people recommending Qwen locally with llama.cpp, they just say "download it" and act like anyone that can't get it running is an idiot. But if there's a "using this settings.json" tutorial somewhere, I didn't find it, and neither did Google over a week of searching.

But Claude got it done for me.

Now, I admit, I haven't played with it much. Just before all this, I ran out of Claude on the $20 plan and bumped up to $100, and It's been so amazing that it's really hard to work on the local. Especially since it feels like Qwen3.8 35b a3b is probably around the corner, and why mess with 3.6 when 3.8 will probably release soon?

With qwen 27b, setting the right reasoning effort for the specific task is important. With xhigh it has a chance at hard problems that bigger models may even fail. But for many everyday tasks, I have found that no reasoning and a system prompt instructing it to be brief is good enough. Note that even with thinking disabled, it may still get into long "chain of thought" reasoning state (out of thinking blocks) if the task is hard and you do not give further instructions, esp with access to tools etc.