Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Why your local LLM feels dumber than it is

forum.level1techs.com

Post reply on HN
Print view View original

181–190 of 233 posts

Re: Why your local LLM feels dumber than it is

#181

Post by vikramkr »

rurban wrote: post #87
jonplackett wrote: post #2

I just got qwen 3.8 27b mlx running on my Macbook Pro and honestly I’m pretty blown away by how not-dumb it is.

We were trying running a local gpt-oss 80GB model on a H100, and honestly I was surprised how dumb it was.

Was there a more recent refresh or is this the model from a year ago? The frontier models were barely functional and almost useless a year ago (gpt oss was pre opus 4.5!) - I would be very surprised if the original drop is anything more than totally obsolete/irrelevant at this point

Re: Why your local LLM feels dumber than it is

#183

Post by catlifeonmars »

embedding-shape wrote: post #157
raffraffraff wrote: post #123

Earlier quoted context omitted.

Wow. I tried to get Qwen3.8 4B to parse song lyrics and analyse them. Getting ollama running was a minute or two. However coming up with a prompt that didn't turn out total garbage was impossible. After wasting over an hour and I ended up getting Qwen side by side with Llama 3.2 3B, just to see if I was being stupid. Nope, it just looks like Llama is orders of magnitude better at this specific task for some reason).…

> Nope, it just looks like Llama is orders of magnitude better at this specific task for some reason This is almost every ML model, if the task isn't part directly or indirectly of the datasets they use for training it, then the model is gonna be pretty trash at it. What the big AI labs have over the smaller labs, is a huge amount of data and diverse set of tasks, hence they generalize better, but still not great. So…

> What the big AI labs have over the smaller labs, is a huge amount of data and diverse set of tasks, hence they generalize better, but still not great.

Hehe, this kind of sounds like the opposite of generalization. As in it’s just specialization at scale.

Re: Why your local LLM feels dumber than it is

#184

Post by shostack »

tarruda wrote: post #144

There's quite a few tangential features that must be implemented correctly or risk affecting the LLM output in significant ways. Parsing/encoding is one example: A couple of months ago I've debugged a reasoning loop bug in Step 3.7 Flash on llama.cpp that was caused by the parser capturing an extra `\n` as part of a reasoning block. It was something that only manifested at longer multi-turn agentic sessions, and the…

Debugging any LLM output when you also have done substantial harness engineering is a total pita and I wish there were better tools for it to isolate issues.

I spent ages tracking down start appears to be an issue with the current Deepseek v4 flash 0731 version that would cause it to output giant walls of gibberish in Hermes with reasoning turned on.

Re: Why your local LLM feels dumber than it is

#185

Post by catlifeonmars »

enraged_camel wrote: post #126
raffraffraff wrote: post #110

Earlier quoted context omitted.

I'm not saying this is the case in the particular example, but these days a lot of people aren't bothering to read a 1 page README, and are instead letting claude "just fucking do it already". I see it at work. Claude has got better at "just fucking doing it" by asking if it's ok to go read the latest github issues and pull the README, which means that people will likely get lazier and lazier.

To me laziness has nothing to do with it. I'd rather focus on my limited time on things that actually matter, e.g. the design of an important product feature or the root cause of a bug and the proper fix. I delegate everything else to Claude. It's really nice to be able to say to Fable "see if you can get this running locally" and then come back 10-30 mins later and read the results. I've discovered a lot of neat too…

FWIW laziness is a strength, not a weakness in software engineering :)

Re: Why your local LLM feels dumber than it is

#186

Post by picofarad »

formerly_proven wrote: post #147
selcuka wrote: post #145

Earlier quoted context omitted.

> There's no official Qwen 3.8 4B (only 27B and 2.4T.. at least for now), so if not a typo you've downloaded some third party model/finetune. Probably this: https://huggingface.co/empero-ai/Qwen3.8-4B-Distill > Qwen3.8-4B is a full-parameter distillation of Qwen3.8 2.4T A95B into the Qwen3.5-4B architecture.

That's not distillation, that's model homeopathy.

Too strong for homeopathy but your point is taken. 0.3% vs 0.000005%

Re: Why your local LLM feels dumber than it is

#187

Post by catlifeonmars »

fxtentacle wrote: post #108
stymaar wrote: post #105

Earlier quoted context omitted.

> It took him two hours of passing errors to Claude for the endpoint to start working What? It's literally three actions and you're good: download llama.cpp, download the model on Huggingface, and run it with. I have no idea how it's supposed to take two hours (unless you have a slow connection and the model download takes this much time, that is).

My experience with Claude is that it suffers badly from “not invented here” syndrome. So probably it rebuilt something like llama from scratch and then 2 hours suddenly seems reasonable (if you don’t question the approach). And that’s the thing, someone with no experience isn’t going to question it.

I wonder if this is an artifact of RL, where the training heavily emphasizes codegen. It may be that the model is just better at generating code than reusing libraries, so it prefers the lowest cost approach.

I also wonder if this manifests much less in contexts where the libraries/frameworks are a large part of the training set. It may be that the model doesn’t generalize well so it’s always better to use knowledge in its training set vs attempting to understand how to use a new, potentially never before seen (from the model perspective) api

Re: Why your local LLM feels dumber than it is

#188

Post by apothegm »

s1gsegv wrote: post #98
velcrovan wrote: post #80

Earlier quoted context omitted.

That's funny, I downloaded the same model on my 48GB M4 Pro and gave it a problem to solve in an existing codebase, it spun its wheels for twenty minutes and then fell over dead. This was using LMStudio and pi as a harness; I never use pi for anything else, so maybe I'm holding it wrong.

They made a kind of strange decision with Qwen3.8 27B, the template defaults the reasoning_effort to xhigh. I found if you set it to medium it doesn’t just sit there churning forever.

Is there an easy way for a n00b with LMStudio to switch it to medium? Asking for a friend… XD

Re: Why your local LLM feels dumber than it is

#189

Post by rurban »

vikramkr wrote: post #181
rurban wrote: post #87

Earlier quoted context omitted.

We were trying running a local gpt-oss 80GB model on a H100, and honestly I was surprised how dumb it was.

Was there a more recent refresh or is this the model from a year ago? The frontier models were barely functional and almost useless a year ago (gpt oss was pre opus 4.5!) - I would be very surprised if the original drop is anything more than totally obsolete/irrelevant at this point

Yes, the old entirely stupid old gpt-oss. But Sonnet and GPT were very useful then already, qwen also.

Re: Why your local LLM feels dumber than it is

#190

Post by formerly_proven »

picofarad wrote: post #186
formerly_proven wrote: post #147

Earlier quoted context omitted.

That's not distillation, that's model homeopathy.

Too strong for homeopathy but your point is taken. 0.3% vs 0.000005%

4B / 2.4T is 1/600 or 0.16%, almost a 3X dilution. A bit on the weak side for homeopathy, but still readily available for many ingredients :)
Post reply on HN