I just got qwen 3.8 27b mlx running on my Macbook Pro and honestly I’m pretty blown away by how not-dumb it is.
How many tok/s are you getting? What gen mbp?
M3 Pro 36GB. I am getting 17 tps with MTPLX.
Why your local LLM feels dumber than it is
51–60 of 233 posts
Re: Why your local LLM feels dumber than it is
#52Even a 4-bit quant of Qwen3.8 27b is indistinguishable from Gemini 3.7 flash in our internal tests. With an RTX5090 card and ninfer, you can get ~800 TPS token generation (c=8) and ~140 Tokens per second single stream.
Re: Why your local LLM feels dumber than it is
#53Earlier quoted context omitted.
My problem is how hot they run. I'm on an m4 pro. Do you have the same issue?
Macs fan control set to full blast and lifting the Mac above table for air circulation across bottom. I use M1 Max with qwen3.8 27b mlx. The gpu temp can reach easily to 95°C as fan doesn’t kick in automatically until 90°C. With Macs fan control at full blast and MBP off table, temp usually hover around 85°C.
Just curious, what is your ram and how many tokens per second do you get with that?
Re: Why your local LLM feels dumber than it is
#54[flagged]
Re: Why your local LLM feels dumber than it is
#55Earlier quoted context omitted.
Macs fan control set to full blast and lifting the Mac above table for air circulation across bottom. I use M1 Max with qwen3.8 27b mlx. The gpu temp can reach easily to 95°C as fan doesn’t kick in automatically until 90°C. With Macs fan control at full blast and MBP off table, temp usually hover around 85°C.
Just curious, what is your ram and how many tokens per second do you get with that?
I am getting like ~25 tps with M5 Max 64GB. Unsloth Desktop and Qwen 3.8 27B Q6 KXL
Re: Why your local LLM feels dumber than it is
#56[flagged]
Re: Why your local LLM feels dumber than it is
#57I just got qwen 3.8 27b mlx running on my Macbook Pro and honestly I’m pretty blown away by how not-dumb it is.
My problem is how hot they run. I'm on an m4 pro. Do you have the same issue?
If you're going to try to use a mac laptop for long term inference you'll probably want to buy one of those $30 laptop fan cooling pads (sold for absurd sized 17 inch x86-64 gaming laptops) on Amazon. Otherwise you'll roast the battery over time and significantly hurt its cycle life and capacity. If you can get significant airflow under the laptop that'll help a lot.
Re: Why your local LLM feels dumber than it is
#58At least I'd be in control of model quality vs. when Anthropic decides to randomly drop the quality of their offering
Re: Why your local LLM feels dumber than it is
#59[flagged]
You've missed a really great human-authored piece then.
Re: Why your local LLM feels dumber than it is
#60most of the time when a local model feels dumb its not the quant, its the chat template. a lot of gguf mints just drop the template from the metadata and the runtime silently falls back to chatml. model still talks fine so nobody notices, it just gets noticeably dumber. got burned by this myself serving qwen, now i grep the gguf for the template tokens before i blame anything else. second place is sampling, people ru…
I've been comparing against TextGen and llama.cpp while I port to LocalAI and have been surprised by what's happening over the API, even with the defaults and jinja. It's been a fair reminder not to eschew familiarizing myself with the repos.
Nothing beats the classic of figuring out something yourself with your brain, but I also like dictating to LLMs a stream of consciousness with what I'm interested in (while forcing it to NOT give any answers or opinions), and getting back file names it suggests I look at and explore.
Modern frontier LLMs can still be used as rubber ducks, and it's a great.