Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Why your local LLM feels dumber than it is

forum.level1techs.com

Post reply on HN
Print view View original

1–10 of 233 posts

Re: Why your local LLM feels dumber than it is

#3

Post by anotherCodder »

most of the time when a local model feels dumb its not the quant, its the chat template. a lot of gguf mints just drop the template from the metadata and the runtime silently falls back to chatml. model still talks fine so nobody notices, it just gets noticeably dumber. got burned by this myself serving qwen, now i grep the gguf for the template tokens before i blame anything else. second place is sampling, people run whatever defaults their ui ships instead of what the vendor recommends and then compare that to benchmark numbers that were run greedy or with the official settings

Re: Why your local LLM feels dumber than it is

#6

Post by downrightmike »

prettyblocks wrote: post #4
jonplackett wrote: post #2

I just got qwen 3.8 27b mlx running on my Macbook Pro and honestly I’m pretty blown away by how not-dumb it is.

My problem is how hot they run. I'm on an m4 pro. Do you have the same issue?

Mineral oil bath?

Re: Why your local LLM feels dumber than it is

#7

Post by lukan »

prettyblocks wrote: post #4
jonplackett wrote: post #2

I just got qwen 3.8 27b mlx running on my Macbook Pro and honestly I’m pretty blown away by how not-dumb it is.

My problem is how hot they run. I'm on an m4 pro. Do you have the same issue?

I don't have the hardware but a often mentioned advice is to put your mac into energy saving mode - it still will work, a bit slower, but stays cool.

Re: Why your local LLM feels dumber than it is

#8

Post by chorlton2080 »

StarlaAtNight wrote: post #5
jonplackett wrote: post #2

I just got qwen 3.8 27b mlx running on my Macbook Pro and honestly I’m pretty blown away by how not-dumb it is.

how quick does it respond? what are specs of your laptop?

Does it need to respond fast? For important applications, I'm sure we'd all be fine waiting 20 minutes for a high quality, usable answer. Or is it the need for interative refinements that make speed relevant?
Post reply on HN