Can you run this on a 36GB MacBook Pro (M3 Pro)? What would be a good setup? for coding mainly
[dead]
Qwen 3.8 27B is excellent, but it defaults to overthinking things
231–240 of 411 posts
Re: Qwen 3.8 27B is excellent, but it defaults to overthinking things
#232Earlier quoted context omitted.
Till now I was using successfully Qwen 3.5 and Gemma 4 at a reasonable speed
There is no way you would run a dense 27b model on that spec. I ran 3.6 27b on a 64gb ram, 24 gb vram, and it felt like the lower limit for this model with a decent context window. If you want a better experience, maybe wait for either a moe model (like 3.6 35b A3) or a model with less parameters (like 9b). Qwen has been releasing those in the past, so maybe we’ll have them for 3.8 too.
From my experience if it doesn't fit on vram it is rarely worth to bother except for a few narrow tasks.
For example make an essay about something where you don't actively engage with the LLM after the initial prompt. So mostly one-shot prompts.
Re: Qwen 3.8 27B is excellent, but it defaults to overthinking things
#233Earlier quoted context omitted.
RAM is never the issue, it's always the compute power
It's absolutely not for these models. There are plenty of consumer GPUs out there with 8 or 12GB VRAM - they are comparatively very fast at inference but just aren't big enough to run lots of the models you want. Also context management is a massive pain.
I run qwen3.5-9B on an RTX 3080 with 10GB of vram. It runs at ~77tk/s with around 50k context size.
As soon as I switch to a model that doesn't fully fit into vram it tanks to <10tk/s which makes it unusable for me for most tasks.
Re: Qwen 3.8 27B is excellent, but it defaults to overthinking things
#234Earlier quoted context omitted.
I removed the extra links to sources for the electricity prices, but that's the average cost in EU, where I live. https://ec.europa.eu/eurostat/web/products-eurostat-news/w/d...
it is not the marginal price for household users, but the (average electricity bill : subscription + consumption) / (average consumption), so it will be higher than the additional kwh consumed.
Maybe, but only it's marginally off.
This is what I pay extra per kWh,in Netherlands: € 0,25162 ($0.291), excluding the connection costs.
Re: Qwen 3.8 27B is excellent, but it defaults to overthinking things
#235Of course, all the latest gains in the latest models are from "thinking" and testing every piece they did.
That's at least my impression. Models didn't get get better, just more thinking and testing and sometimes fixing things you didn't ask for ( hello opus, can you check xxx, opus: I fixed it..)
Next step is a model with 10 GB thinking for ten minutes.
Re: Qwen 3.8 27B is excellent, but it defaults to overthinking things
#236“The fact that a 17GB file can do all of this stuff on my home machines is a miracle. Once again, I’m delighted and amazed at how much progress local models have made this year.” I think that should be the blinking headline - this shows what can be done with consumer hardware.
The other implication here is that this is all software improvements and optimization. There might be a lot more wiggle room for improving quality over time. It seems the model and reasoning quality is improving faster than the hardware currently.
The over reasoning that Simon Willison highlights here is a real issue though. I've observed it with some of the OpenAI models as well. They are prone to overthinking and overengineering things.
What I would love is models that figure out their own appropriate reasoning effort given a task. I'm spending too much brain cycles worrying on what model speed, reasoning, and quality settings to pick. It's not just a cost concern it's also a time concern. Wasting a lot of time for simple UI tweaks because the model is set to high or ultra or whatever is counter productive. The last few iterations of frontier models seem to emphasize benchmarks and reasoning effort.
But of course the day to day reality of many developers is that they are trying to solve relatively simple problems compared to e.g. proving some so far unproven theorems, solving some Nobel prize level problems, etc. I'd love my tools to start making sane choices based on what I ask rather than defaulting to "boil the oceans". These tools need some kind of Auto select. Mostly Ultra is overkill and a waste of time and resources. And of course with local models, keeping simple things local is a nice option.
It's nice to have Sol Ultra extra fast as an option in my back pocket. But it's complete overkill 99% of the time. And it's not like most users make good choices here or are even capable of making good, informed choices. The models are more intelligent than the tool UX. Arguably, a local model of very modest size might be able to do better for this specific choice.
Re: Qwen 3.8 27B is excellent, but it defaults to overthinking things
#237I just used it on a Apple M4 MacBook Pro with 48GB RAM with llama.cpp and Pi to help diagnose an infinite looping request in a React Server component on a Next.js application.
After about 10+ hours of digging, it has apparently found a bug in the Next.js framework, with an example app that replicates the bug, and a fix for now to disable prefetch in the Link component.
I had in my prompt asked it to discover the root cause of the bug and propose a fix, but I did not expect it to dig this deep.
Re: Qwen 3.8 27B is excellent, but it defaults to overthinking things
#238I just used it on a Apple M4 MacBook Pro with 48GB RAM with llama.cpp and Pi to help diagnose an infinite looping request in a React Server component on a Next.js application. After about 10+ hours of digging, it has apparently found a bug in the Next.js framework, with an example app that replicates the bug, and a fix for now to disable prefetch in the Link component. I had in my prompt asked it to discover the root…
I am eyeing one of these specifically for this use case, could you please post roughly what kind of tokens per second numbers you get for text generation for this 27B model?
edit: and which quant you are using, please :-)
Re: Qwen 3.8 27B is excellent, but it defaults to overthinking things
#239I just used it on a Apple M4 MacBook Pro with 48GB RAM with llama.cpp and Pi to help diagnose an infinite looping request in a React Server component on a Next.js application. After about 10+ hours of digging, it has apparently found a bug in the Next.js framework, with an example app that replicates the bug, and a fix for now to disable prefetch in the Link component. I had in my prompt asked it to discover the root…
10 hours at what tokens per sec?
Re: Qwen 3.8 27B is excellent, but it defaults to overthinking things
#240I truly hope we get a Qwen 3.8 35B-A3B
The model ID appeared in some alibaba PR but later disappeared.
It's the optimal blend of accessibility and model size for a lot of people.