And here i am with a unsloth UD Q4_K_XL quant. Its a good model still.
Why your local LLM feels dumber than it is
121–130 of 233 posts
Re: Why your local LLM feels dumber than it is
#122Earlier quoted context omitted.
I'm not saying this is the case in the particular example, but these days a lot of people aren't bothering to read a 1 page README, and are instead letting claude "just fucking do it already". I see it at work. Claude has got better at "just fucking doing it" by asking if it's ok to go read the latest github issues and pull the README, which means that people will likely get lazier and lazier.
[flagged]
Add Just [...] And you have also fully adopted the writing patterns.
Re: Why your local LLM feels dumber than it is
#123I saw some guy streaming how he was deploying qwen3.8 37B on his local setup. Well, he was asking Claude to do it. It took him two hours of passing errors to Claude for the endpoint to start working, he then started testing it against DS4 Flash when Qwen had thinking disabled and Claude messed up sampling parameters, it was an absolute pain to watch
Wow. I tried to get Qwen3.8 4B to parse song lyrics and analyse them. Getting ollama running was a minute or two.
However coming up with a prompt that didn't turn out total garbage was impossible. After wasting over an hour and I ended up getting Qwen side by side with Llama 3.2 3B, just to see if I was being stupid. Nope, it just looks like Llama is orders of magnitude better at this specific task for some reason).
If you think I'm doing it wrong, you're probably right, I don't know a ton about local LLMs. But I hand selected 50 songs, set up ollama with both LLMs, and for each iteration on the prompt text, ran both LLMs 10x times per song. Side-by-side comparisons showed that Qwen 3 4B was so bad that I actually downloaded Qwen again, thinking there must have been some mistake and I accidentally grabbed an old 1B model.
Re: Why your local LLM feels dumber than it is
#124Re: Why your local LLM feels dumber than it is
#125Earlier quoted context omitted.
My problem is how hot they run. I'm on an m4 pro. Do you have the same issue?
It’s hot and also LOUD and runs the battery down quick. But I’m having a lot of luck just running things when I’m away from the computer and can leave it plugged in. It starts going weird (unreliable and slow) with context over 80k so you have to pick tasks one at a time and baby sit a lot more than Claude. But it really is very capable and feels like there’s an intelligence there to talk to. Maybe gpt-4 level clever…
If you're running it while idle and don't need the quickest results, reducing the clock speed improves energy efficiency (and in your case avoids overheating the battery). There will be some optimal speed that maximizes computations per joule that depends on the specific load and can only be found by measurement.
On Linux, you can cap CPU frequencies with "cpupower". Does MacOS have any equivalent?
Re: Why your local LLM feels dumber than it is
#126Earlier quoted context omitted.
> It took him two hours of passing errors to Claude for the endpoint to start working What? It's literally three actions and you're good: download llama.cpp, download the model on Huggingface, and run it with. I have no idea how it's supposed to take two hours (unless you have a slow connection and the model download takes this much time, that is).
I'm not saying this is the case in the particular example, but these days a lot of people aren't bothering to read a 1 page README, and are instead letting claude "just fucking do it already". I see it at work. Claude has got better at "just fucking doing it" by asking if it's ok to go read the latest github issues and pull the README, which means that people will likely get lazier and lazier.
To me laziness has nothing to do with it. I'd rather focus on my limited time on things that actually matter, e.g. the design of an important product feature or the root cause of a bug and the proper fix. I delegate everything else to Claude. It's really nice to be able to say to Fable "see if you can get this running locally" and then come back 10-30 mins later and read the results.
I've discovered a lot of neat tooling this way that I otherwise would not have bothered tryin to set up because that can take up a lot of time. And even when it's fast, you're suffering from context-switching penalties. You framed it as "read the latest github issues and pull the README" but anyone who has worked in tech for an appreciable amount of time knows that that stuff can turn into a deep rabbit hole.
Re: Why your local LLM feels dumber than it is
#127I just got qwen 3.8 27b mlx running on my Macbook Pro and honestly I’m pretty blown away by how not-dumb it is.
That's funny, I downloaded the same model on my 48GB M4 Pro and gave it a problem to solve in an existing codebase, it spun its wheels for twenty minutes and then fell over dead. This was using LMStudio and pi as a harness; I never use pi for anything else, so maybe I'm holding it wrong.
Set its thinking lower. This is a known issue. It still thinks A LOT with lower reasoning levels
Re: Why your local LLM feels dumber than it is
#128Earlier quoted context omitted.
It’s hot and also LOUD and runs the battery down quick. But I’m having a lot of luck just running things when I’m away from the computer and can leave it plugged in. It starts going weird (unreliable and slow) with context over 80k so you have to pick tasks one at a time and baby sit a lot more than Claude. But it really is very capable and feels like there’s an intelligence there to talk to. Maybe gpt-4 level clever…
If you're running it while idle and don't need the quickest results, reducing the clock speed improves energy efficiency (and in your case avoids overheating the battery). There will be some optimal speed that maximizes computations per joule that depends on the specific load and can only be found by measurement. On Linux, you can cap CPU frequencies with "cpupower". Does MacOS have any equivalent?
The Mac unfortunately just has two performance modes ‘all out power and melting’ or ‘cold and really really slow’
Re: Why your local LLM feels dumber than it is
#129Earlier quoted context omitted.
It’s hot and also LOUD and runs the battery down quick. But I’m having a lot of luck just running things when I’m away from the computer and can leave it plugged in. It starts going weird (unreliable and slow) with context over 80k so you have to pick tasks one at a time and baby sit a lot more than Claude. But it really is very capable and feels like there’s an intelligence there to talk to. Maybe gpt-4 level clever…
> It’s hot and also LOUD and runs the battery down quick. Who'd run this on battery? Or do you mean kills the battery faster even when used while charging?
I was trying to do some work on the train with no signal.
Re: Why your local LLM feels dumber than it is
#130Earlier quoted context omitted.
They made a kind of strange decision with Qwen3.8 27B, the template defaults the reasoning_effort to xhigh. I found if you set it to medium it doesn’t just sit there churning forever.
xhigh gives better results
Not necessarily.
I have seen xhigh go down several rabbit holes, dwell on edge cases and write worse code as a result; it literally distracted itself into writing a complex chain of functions ignoring my prompt, when on “low” reasoning it gets it right on a prompt that requires a few lines of code in the right places.
Simon Willison’s blog has another example (SVG of a circle).
It’s a bit like how giving LLMs access to web search tools can cause them to go down a blind alley based on their first “reasoning” output that then leaves them unable to solve a puzzle correctly that they can fully solve on their own.