Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Why your local LLM feels dumber than it is

forum.level1techs.com

Post reply on HN
Print view View original

131–140 of 233 posts

Re: Why your local LLM feels dumber than it is

#131

Post by petu »

raffraffraff wrote: post #123
big-chungus4 wrote: post #102

I saw some guy streaming how he was deploying qwen3.8 37B on his local setup. Well, he was asking Claude to do it. It took him two hours of passing errors to Claude for the endpoint to start working, he then started testing it against DS4 Flash when Qwen had thinking disabled and Claude messed up sampling parameters, it was an absolute pain to watch

Wow. I tried to get Qwen3.8 4B to parse song lyrics and analyse them. Getting ollama running was a minute or two. However coming up with a prompt that didn't turn out total garbage was impossible. After wasting over an hour and I ended up getting Qwen side by side with Llama 3.2 3B, just to see if I was being stupid. Nope, it just looks like Llama is orders of magnitude better at this specific task for some reason).…

There's no official Qwen 3.8 4B (only 27B and 2.4T.. at least for now), so if not a typo you've downloaded some third party model/finetune.

Also if you have less than 24GB VRAM, then ollama defaults to 4K context. If that "Qwen 3.8" uses thinking, it might be running out of context and forgetting what it was even answering mid-generation. If that's the case, then try increasing context size: https://docs.ollama.com/context-length , but also: https://sleepingrobots.com/dreams/stop-using-ollama/

Re: Why your local LLM feels dumber than it is

#132

Post by LeBit »

stymaar wrote: post #105
big-chungus4 wrote: post #102

I saw some guy streaming how he was deploying qwen3.8 37B on his local setup. Well, he was asking Claude to do it. It took him two hours of passing errors to Claude for the endpoint to start working, he then started testing it against DS4 Flash when Qwen had thinking disabled and Claude messed up sampling parameters, it was an absolute pain to watch

> It took him two hours of passing errors to Claude for the endpoint to start working What? It's literally three actions and you're good: download llama.cpp, download the model on Huggingface, and run it with. I have no idea how it's supposed to take two hours (unless you have a slow connection and the model download takes this much time, that is).

In the README there is usually a section where it explains which llama.cpp parameters to configure (sampling, penalties, context size, etc) to get better results.

Re: Why your local LLM feels dumber than it is

#133

Post by LeBit »

LeifCarrotson wrote: post #69
coldtea wrote: post #61

Earlier quoted context omitted.

> It’s hot and also LOUD and runs the battery down quick. Who'd run this on battery? Or do you mean kills the battery faster even when used while charging?

There was a lovely window of a few years when processors were fast enough and low-power enough that real development work could trivially happen on a Macbook Air in a lounge. That window is waning as more and more memory and graphics processing power is being used locally. I suppose the future is most likely going to involve farming out AI requests to your desktop machine, your company's compute farm, or a cloud endp…

Running your dev environment on your laptop is so 2025.

Re: Why your local LLM feels dumber than it is

#134

Post by LeBit »

petcat wrote: post #35
dominotw wrote: post #18

Earlier quoted context omitted.

i suspect ppl dropping generic "its awesome" comments are not actually using it and prbly just managed to get it running for a prompt or two.

Yeah, that's my experience. It's a big "wow" factor to get a non-trivial LLM running on my Mac, but it's actually not that useful. Like trying to use Photoshop at 8 FPS.

It’s not because you didn’t find use cases for local LLMs that there are none.

I use local LLMs on my Mac Mini M4 Pro with 48G to review text messages tone, act as a text correction tool, act as a code review tool, to do code agent work, generate code snippets, etc

Gemma 4 26B A4B gives me steady 20 tps.

Re: Why your local LLM feels dumber than it is

#135

Post by digikata »

mistersquid wrote: post #68
dominotw wrote: post #18

Earlier quoted context omitted.

i suspect ppl dropping generic "its awesome" comments are not actually using it and prbly just managed to get it running for a prompt or two.

Seems threads about local LLMs on Apple hardware feature comments listing M3/4/5 at 48GB 64GB and not 128GB. That is, users with M-series hardware that have less-than-max RAM share results whereas users with max RAM do not. Speculating (not extrapolating), maybe users with machine that have max RAM are less interested in running local LLMs and are less averse to paying services for compute? Personally, I’d love to se…

I've been using Qwen3.6-37B-A3B on an M1 Max w/ llama.cpp and for my practical uses I prefer it to qwen3.8. When 3.8 does answer its slower and, qualitatively, marginally better than qwen3.6, but 3.8 often ends up in unresolved thought loops and runs slower. The Moe 3.6 on my setup is much faster, 500t/s peaks, 30t/s typical, vs 3.8 150 peak, 4-9 t/s typical.

While I've spend a little time tuning, I'm assuming there will be deeper tuning for 3.8 that might close the gap.

Re: Why your local LLM feels dumber than it is

#136

Post by chmod775 »

ekianjo wrote: post #101
s1gsegv wrote: post #98

Earlier quoted context omitted.

They made a kind of strange decision with Qwen3.8 27B, the template defaults the reasoning_effort to xhigh. I found if you set it to medium it doesn’t just sit there churning forever.

xhigh gives better results

Not if it fills up its entire context with "But wait..."

Re: Why your local LLM feels dumber than it is

#137

Post by chmod775 »

adrianN wrote: post #74
coldtea wrote: post #61

Earlier quoted context omitted.

> It’s hot and also LOUD and runs the battery down quick. Who'd run this on battery? Or do you mean kills the battery faster even when used while charging?

Running the laptop hot ages the battery even when you’re plugged in.

Merely existing noticeably ages most battery chemistries.

You can take half an hour out of every week inconveniencing yourself to protect your battery, or you can spend an hour once a year just putting in a new one (or having it done for you in-store if you're using an Apple device) and save yourself the nagging worry.

Re: Why your local LLM feels dumber than it is

#139

Post by Abishek_Muthian »

After 3 years of running local models I think the model which fits unquantized (BF16) in the VRAM is the best model for general purpose tasks; fine-tuned SLMs or utilities based on non language models for solving a specific problem (e.g. TTS,STT,RMBG etc.) have been the best use of local AI for me.

Local models for coding, is just not worth the effort IMO; unless of course you have the hardware to fit it unquantized in your VRAM.

Re: Why your local LLM feels dumber than it is

#140

Post by 6wf9j »

raffraffraff wrote: post #123
big-chungus4 wrote: post #102

I saw some guy streaming how he was deploying qwen3.8 37B on his local setup. Well, he was asking Claude to do it. It took him two hours of passing errors to Claude for the endpoint to start working, he then started testing it against DS4 Flash when Qwen had thinking disabled and Claude messed up sampling parameters, it was an absolute pain to watch

Wow. I tried to get Qwen3.8 4B to parse song lyrics and analyse them. Getting ollama running was a minute or two. However coming up with a prompt that didn't turn out total garbage was impossible. After wasting over an hour and I ended up getting Qwen side by side with Llama 3.2 3B, just to see if I was being stupid. Nope, it just looks like Llama is orders of magnitude better at this specific task for some reason).…

I found Gemma 4 models quite good at translating/explaining song lyrics. I actually prefer using 26B-A4B for this task instead of GPT
Post reply on HN