Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Why your local LLM feels dumber than it is

forum.level1techs.com

Post reply on HN
Print view View original

191–200 of 233 posts

Re: Why your local LLM feels dumber than it is

#191

Post by BoredomIsFun »

raffraffraff wrote: post #123
big-chungus4 wrote: post #102

I saw some guy streaming how he was deploying qwen3.8 37B on his local setup. Well, he was asking Claude to do it. It took him two hours of passing errors to Claude for the endpoint to start working, he then started testing it against DS4 Flash when Qwen had thinking disabled and Claude messed up sampling parameters, it was an absolute pain to watch

Wow. I tried to get Qwen3.8 4B to parse song lyrics and analyse them. Getting ollama running was a minute or two. However coming up with a prompt that didn't turn out total garbage was impossible. After wasting over an hour and I ended up getting Qwen side by side with Llama 3.2 3B, just to see if I was being stupid. Nope, it just looks like Llama is orders of magnitude better at this specific task for some reason).…

Llama 3.2 3B was surprisingly good for general-purpose text manipulation tasks. I'd argue it might be better than many modern tiny models for that.

Re: Why your local LLM feels dumber than it is

#192

Post by ohyes »

wccrawford wrote: post #148
big-chungus4 wrote: post #102

I saw some guy streaming how he was deploying qwen3.8 37B on his local setup. Well, he was asking Claude to do it. It took him two hours of passing errors to Claude for the endpoint to start working, he then started testing it against DS4 Flash when Qwen had thinking disabled and Claude messed up sampling parameters, it was an absolute pain to watch

That's funny, because I just went through the opposite. I got llama.cpp working with qwen3.6 and qwen3.8 by Googling and manually adjusting things according to reddit posts and Google not-really-helpful AI suggestions. I tried settings up per-model stuff in settings.json, but again Google got in my way, and llama.cpp having 2 different settings.json (and Google lying about where 1 goes) made it far too difficulty to…

Am I the only one who just downloads directly from LM Studio and just runs the server there? It’s trivial.

Re: Why your local LLM feels dumber than it is

#193

Post by meroes »

catlifeonmars wrote: post #183
embedding-shape wrote: post #157

Earlier quoted context omitted.

> Nope, it just looks like Llama is orders of magnitude better at this specific task for some reason This is almost every ML model, if the task isn't part directly or indirectly of the datasets they use for training it, then the model is gonna be pretty trash at it. What the big AI labs have over the smaller labs, is a huge amount of data and diverse set of tasks, hence they generalize better, but still not great. So…

> What the big AI labs have over the smaller labs, is a huge amount of data and diverse set of tasks, hence they generalize better, but still not great. Hehe, this kind of sounds like the opposite of generalization. As in it’s just specialization at scale.

It is specialization at scale.

It’s paying hundreds of thousands of RLHF’ers from every subject and through some dystopian income stream.

It’s decent money don’t get me wrong, but you aren’t paid at if a task isn’t completed in time for example.

Re: Why your local LLM feels dumber than it is

#194

Post by velcrovan »

s1gsegv wrote: post #98
velcrovan wrote: post #80

Earlier quoted context omitted.

That's funny, I downloaded the same model on my 48GB M4 Pro and gave it a problem to solve in an existing codebase, it spun its wheels for twenty minutes and then fell over dead. This was using LMStudio and pi as a harness; I never use pi for anything else, so maybe I'm holding it wrong.

They made a kind of strange decision with Qwen3.8 27B, the template defaults the reasoning_effort to xhigh. I found if you set it to medium it doesn’t just sit there churning forever.

I had heard of this and actually did set the reasoning to medium ahead of time…

Re: Why your local LLM feels dumber than it is

#195

Post by runeks »

Zylokloto wrote: post #160
runeks wrote: post #152

Somewhat off topic, but I've started wondering if we can actually make local LLMs feel smarter than the frontier closed models, by post-training it for your specific use case. Say Company X has a software product which consists of a million lines of code, including a ticket for every bug and new feature for this piece of software. Then wouldn't it make sense to try using an open weight model but post-train it on that…

You could do this for sure but its quite a lot of effort and a normal software stack is quite well known due to the massive amount of github projects and other opensource code. You can also get a lot out of a harness in this case or your Agents.md or Claude.md file by just enhancing the context. You might even question yourself if you are doing something wrong if a modern LLM really struggles with your code. We do th…

> We do the finetuning only on small semantic data were it helps a lot.

This sounds interesting. Would you care to expand a bit on how you do this? What is this semantic data?

> There are companies though which ahve this exact problem with programming languages you normally don't see.

This applies to my company. We have a substantial code base in our own dialect of APL. You think post-training would help substantially here?

Re: Why your local LLM feels dumber than it is

#196

Post by runeks »

rovr138 wrote: post #166
runeks wrote: post #152

Somewhat off topic, but I've started wondering if we can actually make local LLMs feel smarter than the frontier closed models, by post-training it for your specific use case. Say Company X has a software product which consists of a million lines of code, including a ticket for every bug and new feature for this piece of software. Then wouldn't it make sense to try using an open weight model but post-train it on that…

Instead, I'd build tooling for the model to be able to query the tickets, pr, commits, diff, etc This is something I can reuse better.

I'd like to have a model that just "knows" our code base and past issues, instead of having to query them for the same reason I don't want the model to have to query a dictionary to speak proper English.

I could be mistaken here, but I'm curious to see if it would improve both output speed and quality. Currently, every new prompt I start it basically spends 10-15 minutes "getting familiar with the code" which is both annoying to wait for and wasteful money-wise.

Re: Why your local LLM feels dumber than it is

#197

Post by tharkun__ »

aktenlage wrote: post #106
tharkun__ wrote: post #72

Earlier quoted context omitted.

It was actually great. I have like a non-AI box so to speak 8GB VRAM, co-incidentally from a gaming PC ... All the previous models that were "frontier level, just try it!" but wouldn't run at all in agentic mode, including previous Qwens, just disappointed, period. Then I ran then Qwen 3.8 27b and while it was super slow (4t/s) it literally one-shotted creating a usable "web search/pull" skill for `pi.dev` . while an…

Have you tried a mixture of experts model? Dense models have been quite slow for me, as I have only 6 GB VRAM. But with llama.cpp and --cpu-moe I get 200 t/s input and almost 30 t/s output with Gemma 4 26B A3B, which feels ok to use. Would be interested about your mileage there.

If I use the 12B Unified (dense) model I mentioned without MTP, then I get 37t/s, input ~700t/s.

It's all still quite frustrating in the end, like a Claude from a very long time ago by now but usable. If I want 64k context, I can't use MTP. I still haven't decided whether I'd rather have 37t/s but it's "less dumb" or I want MTP speed but it's going off the rails more. All of this is also with `-ctv q4_0 -ctk q4_0)`, which is not ideal. I'm actually right now contending with 35k context but using q8_0 KV quantization. More like 35t/s coz with those settings I can't use MTP.

But I'm not ready to go back to 4t/s. It's not interactive enough for me. That said, I had tried to use the Gemma E4B for example to have it build itself that websearch/fetch skill. It utterly failed, as did previous qwens.

I don't see a Gemma 4 26B A3B GGUF for download, but there is a gemma-4-26B-A4B-it-MXFP4_MOE.gguf that should fit into my overall RAM and then use lots of CPU like the Qwen 3.8. I guess I'll give it a try just to see the difference in speed though I don't expect anything "usable" out of that tbh.

Re: Why your local LLM feels dumber than it is

#198

Post by tsukikage »

tarruda wrote: post #144

There's quite a few tangential features that must be implemented correctly or risk affecting the LLM output in significant ways. Parsing/encoding is one example: A couple of months ago I've debugged a reasoning loop bug in Step 3.7 Flash on llama.cpp that was caused by the parser capturing an extra `\n` as part of a reasoning block. It was something that only manifested at longer multi-turn agentic sessions, and the…

Meanwhile, entirely too frequently I see unit tests that have atol / rtol values which permit errors of one part in 200 or even higher, where people bother with unit tests at all. At some point someone in the pytorch ecosystem used rtol=5e-03 in a unit test and crowds of people started passing that around unquestioned.

With attention matrix sizes being what they are, that's high enough that you can literally zero out a row or two and still have the test suite pass. Guess how I know!

Check your numbers, folks.

Re: Why your local LLM feels dumber than it is

#199

Post by stillpointlab »

catlifeonmars wrote: post #180
tarruda wrote: post #144

There's quite a few tangential features that must be implemented correctly or risk affecting the LLM output in significant ways. Parsing/encoding is one example: A couple of months ago I've debugged a reasoning loop bug in Step 3.7 Flash on llama.cpp that was caused by the parser capturing an extra `\n` as part of a reasoning block. It was something that only manifested at longer multi-turn agentic sessions, and the…

This is fascinating. I’m struggling to understand how that was causing such a large difference in the output. Is the “autoparser” vulnerable to injections somehow? How do you distinguish between user text, model text, and metadata, or is there ambiguity in the parsing?

My take, as someone who just read through the github issues conversation.

The model was outputting reasoning traces that were supposed to lead to tool calls. So the model might do something like:

    
    I should use a tool
    

    ... should make the tool call here
But a \n was slipping through from the last line of the reasoning trace so the parser was generating:

   
   I should use a tool
And that extra new line before the closing would occasionally trigger the model to question itself with an "Actually ... " digression. In long running conversations this would end up looping because the "Actually ..." part would reason it should call a tool, then a new trailing \n would trigger an "Actually ..." and then it ends up in a loop.
Post reply on HN