Why your local LLM feels dumber than it is • Hacker News Board

Re: Why your local LLM feels dumber than it is

#61

Post by coldtea » Sun, Aug 23, 2026, 1:18 AM UTC

jonplackett wrote: post #13

prettyblocks wrote: post #4

Earlier quoted context omitted.

My problem is how hot they run. I'm on an m4 pro. Do you have the same issue?

It’s hot and also LOUD and runs the battery down quick. But I’m having a lot of luck just running things when I’m away from the computer and can leave it plugged in. It starts going weird (unreliable and slow) with context over 80k so you have to pick tasks one at a time and baby sit a lot more than Claude. But it really is very capable and feels like there’s an intelligence there to talk to. Maybe gpt-4 level clever…

>It’s hot and also LOUD and runs the battery down quick.

Who'd run this on battery?

Or do you mean kills the battery faster even when used while charging?

Re: Why your local LLM feels dumber than it is

#62

Post by coldtea » Sun, Aug 23, 2026, 1:21 AM UTC

petcat wrote: post #35

dominotw wrote: post #18

Earlier quoted context omitted.

i suspect ppl dropping generic "its awesome" comments are not actually using it and prbly just managed to get it running for a prompt or two.

Yeah, that's my experience. It's a big "wow" factor to get a non-trivial LLM running on my Mac, but it's actually not that useful. Like trying to use Photoshop at 8 FPS.

Regarding this analogy, fps don't matter as much for Photoshop, since it's not an immediate mode GUI. 8 fps would be quite ok for comfortably getting feedback on live image filters and such.

Re: Why your local LLM feels dumber than it is

#63

Post by fenestella » Sun, Aug 23, 2026, 1:32 AM UTC

The section on system prompts and context window management is spot on; most people don't realize how much the default quantization in popular runners degrades logic compared to full FP16. I'd be curious to see if the author has benchmarked the impact of KV cache compression on longer context reasoning, as that usually seems to be where my local Llama 3 setup starts to fall apart.

Re: Why your local LLM feels dumber than it is

#64

Post by throwdbaaway » Sun, Aug 23, 2026, 1:35 AM UTC

> Both the NVFP4 and AWQ W4A16 failed to properly close their tool calls ...

If I understand correctly, this failure mode is just not possible with llama.cpp / ik_llama.cpp, which enforces token generation to follow the grammar once a tool call is detected.

> ... and botched Cisco command line syntax (the correct command was ‘show arp’, while they executed ‘show run’)

But this failure mode can still happen.

Anyway, NVFP4 and AWQ W4A16 are generally regarded as low quality quants. IQK/Trellis quants from ik_llama.cpp and EXL3 quants from exllama should work better.

So, perhaps the lesson here is "don't use vllm at home"?

Re: Why your local LLM feels dumber than it is

#65

Post by Forgeties79 » Sun, Aug 23, 2026, 1:41 AM UTC

akg_67 wrote: post #46

prettyblocks wrote: post #4

Earlier quoted context omitted.

My problem is how hot they run. I'm on an m4 pro. Do you have the same issue?

Macs fan control set to full blast and lifting the Mac above table for air circulation across bottom. I use M1 Max with qwen3.8 27b mlx. The gpu temp can reach easily to 95°C as fan doesn’t kick in automatically until 90°C. With Macs fan control at full blast and MBP off table, temp usually hover around 85°C.

Yikes isn’t 95 well above the danger zone for most components or have things changed? I remember crypto mining in the early 2010’s and making sure to stay under 85C on my GPU.

Re: Why your local LLM feels dumber than it is

#66

Post by throwdbaaway » Sun, Aug 23, 2026, 1:42 AM UTC

throwdbaaway wrote: post #64

> Both the NVFP4 and AWQ W4A16 failed to properly close their tool calls ... If I understand correctly, this failure mode is just not possible with llama.cpp / ik_llama.cpp, which enforces token generation to follow the grammar once a tool call is detected. > ... and botched Cisco command line syntax (the correct command was ‘show arp’, while they executed ‘show run’) But this failure mode can still happen. Anyway, N…

As for KV cache quantization, Q8_0 from llama.cpp / ik_llama.cpp should also work better than FP8 from vllm (see https://github.com/vllm-project/vllm/issues/33480#issuecomme...).

Re: Why your local LLM feels dumber than it is

#67

Post by akg_67 » Sun, Aug 23, 2026, 1:43 AM UTC

Footprint0521 wrote: post #53

akg_67 wrote: post #46

Earlier quoted context omitted.

Macs fan control set to full blast and lifting the Mac above table for air circulation across bottom. I use M1 Max with qwen3.8 27b mlx. The gpu temp can reach easily to 95°C as fan doesn’t kick in automatically until 90°C. With Macs fan control at full blast and MBP off table, temp usually hover around 85°C.

Just curious, what is your ram and how many tokens per second do you get with that?

RAM 32GB, tokens are less than 20, may be about 12-15. Main issue is context window, I have set to 64k but practically beyond 32k, too much swap and slow down.

Re: Why your local LLM feels dumber than it is

#68

Post by mistersquid » Sun, Aug 23, 2026, 1:48 AM UTC

dominotw wrote: post #18

alexchantavy wrote: post #9

Earlier quoted context omitted.

How many tok/s are you getting? What gen mbp?

i suspect ppl dropping generic "its awesome" comments are not actually using it and prbly just managed to get it running for a prompt or two.

Seems threads about local LLMs on Apple hardware feature comments listing M3/4/5 at 48GB 64GB and not 128GB.

That is, users with M-series hardware that have less-than-max RAM share results whereas users with max RAM do not.

Speculating (not extrapolating), maybe users with machine that have max RAM are less interested in running local LLMs and are less averse to paying services for compute?

Personally, I’d love to see what output max RAM M-series Apple hardware in these threads.

Re: Why your local LLM feels dumber than it is

#69

Post by LeifCarrotson » Sun, Aug 23, 2026, 1:55 AM UTC

coldtea wrote: post #61

jonplackett wrote: post #13

Earlier quoted context omitted.

It’s hot and also LOUD and runs the battery down quick. But I’m having a lot of luck just running things when I’m away from the computer and can leave it plugged in. It starts going weird (unreliable and slow) with context over 80k so you have to pick tasks one at a time and baby sit a lot more than Claude. But it really is very capable and feels like there’s an intelligence there to talk to. Maybe gpt-4 level clever…

> It’s hot and also LOUD and runs the battery down quick. Who'd run this on battery? Or do you mean kills the battery faster even when used while charging?

There was a lovely window of a few years when processors were fast enough and low-power enough that real development work could trivially happen on a Macbook Air in a lounge.

That window is waning as more and more memory and graphics processing power is being used locally.

I suppose the future is most likely going to involve farming out AI requests to your desktop machine, your company's compute farm, or a cloud endpoint, but if you're expecting to run an AI offline on your lap with a battery...your lap will get warm.

Re: Why your local LLM feels dumber than it is

#70

Post by riddlemethat » Sun, Aug 23, 2026, 1:58 AM UTC

jonplackett wrote: post #2

I just got qwen 3.8 27b mlx running on my Macbook Pro and honestly I’m pretty blown away by how not-dumb it is.

I got the qwen 3.8 abliterated model running on my MacBook Pro M5 48GB and it's pretty nice having a local model that can do a lot of experimentation without rails.