Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Why your local LLM feels dumber than it is

forum.level1techs.com

Post reply on HN
Print view View original

101–110 of 233 posts

Re: Why your local LLM feels dumber than it is

#101

Post by ekianjo »

s1gsegv wrote: post #98
velcrovan wrote: post #80

Earlier quoted context omitted.

That's funny, I downloaded the same model on my 48GB M4 Pro and gave it a problem to solve in an existing codebase, it spun its wheels for twenty minutes and then fell over dead. This was using LMStudio and pi as a harness; I never use pi for anything else, so maybe I'm holding it wrong.

They made a kind of strange decision with Qwen3.8 27B, the template defaults the reasoning_effort to xhigh. I found if you set it to medium it doesn’t just sit there churning forever.

xhigh gives better results

Re: Why your local LLM feels dumber than it is

#102

Post by big-chungus4 »

I saw some guy streaming how he was deploying qwen3.8 37B on his local setup. Well, he was asking Claude to do it. It took him two hours of passing errors to Claude for the endpoint to start working, he then started testing it against DS4 Flash when Qwen had thinking disabled and Claude messed up sampling parameters, it was an absolute pain to watch

Re: Why your local LLM feels dumber than it is

#105

Post by stymaar »

big-chungus4 wrote: post #102

I saw some guy streaming how he was deploying qwen3.8 37B on his local setup. Well, he was asking Claude to do it. It took him two hours of passing errors to Claude for the endpoint to start working, he then started testing it against DS4 Flash when Qwen had thinking disabled and Claude messed up sampling parameters, it was an absolute pain to watch

> It took him two hours of passing errors to Claude for the endpoint to start working

What? It's literally three actions and you're good: download llama.cpp, download the model on Huggingface, and run it with.

I have no idea how it's supposed to take two hours (unless you have a slow connection and the model download takes this much time, that is).

Re: Why your local LLM feels dumber than it is

#106

Post by aktenlage »

tharkun__ wrote: post #72
jonplackett wrote: post #2

I just got qwen 3.8 27b mlx running on my Macbook Pro and honestly I’m pretty blown away by how not-dumb it is.

It was actually great. I have like a non-AI box so to speak 8GB VRAM, co-incidentally from a gaming PC ... All the previous models that were "frontier level, just try it!" but wouldn't run at all in agentic mode, including previous Qwens, just disappointed, period. Then I ran then Qwen 3.8 27b and while it was super slow (4t/s) it literally one-shotted creating a usable "web search/pull" skill for `pi.dev` . while an…

Have you tried a mixture of experts model? Dense models have been quite slow for me, as I have only 6 GB VRAM. But with llama.cpp and --cpu-moe I get 200 t/s input and almost 30 t/s output with Gemma 4 26B A3B, which feels ok to use. Would be interested about your mileage there.

Re: Why your local LLM feels dumber than it is

#107

Post by Schiendelman »

stymaar wrote: post #105
big-chungus4 wrote: post #102

I saw some guy streaming how he was deploying qwen3.8 37B on his local setup. Well, he was asking Claude to do it. It took him two hours of passing errors to Claude for the endpoint to start working, he then started testing it against DS4 Flash when Qwen had thinking disabled and Claude messed up sampling parameters, it was an absolute pain to watch

> It took him two hours of passing errors to Claude for the endpoint to start working What? It's literally three actions and you're good: download llama.cpp, download the model on Huggingface, and run it with. I have no idea how it's supposed to take two hours (unless you have a slow connection and the model download takes this much time, that is).

Your mileage may vary. I tried this a couple months ago and spent a full day on it just not working before giving up. Anything I sent, it wouldn't run.

Re: Why your local LLM feels dumber than it is

#108

Post by fxtentacle »

stymaar wrote: post #105
big-chungus4 wrote: post #102

I saw some guy streaming how he was deploying qwen3.8 37B on his local setup. Well, he was asking Claude to do it. It took him two hours of passing errors to Claude for the endpoint to start working, he then started testing it against DS4 Flash when Qwen had thinking disabled and Claude messed up sampling parameters, it was an absolute pain to watch

> It took him two hours of passing errors to Claude for the endpoint to start working What? It's literally three actions and you're good: download llama.cpp, download the model on Huggingface, and run it with. I have no idea how it's supposed to take two hours (unless you have a slow connection and the model download takes this much time, that is).

My experience with Claude is that it suffers badly from “not invented here” syndrome. So probably it rebuilt something like llama from scratch and then 2 hours suddenly seems reasonable (if you don’t question the approach). And that’s the thing, someone with no experience isn’t going to question it.

Re: Why your local LLM feels dumber than it is

#110

Post by raffraffraff »

stymaar wrote: post #105
big-chungus4 wrote: post #102

I saw some guy streaming how he was deploying qwen3.8 37B on his local setup. Well, he was asking Claude to do it. It took him two hours of passing errors to Claude for the endpoint to start working, he then started testing it against DS4 Flash when Qwen had thinking disabled and Claude messed up sampling parameters, it was an absolute pain to watch

> It took him two hours of passing errors to Claude for the endpoint to start working What? It's literally three actions and you're good: download llama.cpp, download the model on Huggingface, and run it with. I have no idea how it's supposed to take two hours (unless you have a slow connection and the model download takes this much time, that is).

I'm not saying this is the case in the particular example, but these days a lot of people aren't bothering to read a 1 page README, and are instead letting claude "just fucking do it already". I see it at work.

Claude has got better at "just fucking doing it" by asking if it's ok to go read the latest github issues and pull the README, which means that people will likely get lazier and lazier.

Post reply on HN