So if I have to set this up on my 24GB MBP what'd the right configuration and tuning look like?
Qwen 3.8 27B is excellent, but it defaults to overthinking things
71–80 of 411 posts
Re: Qwen 3.8 27B is excellent, but it defaults to overthinking things
#72All current era models overthink as it's a product of their RL incentives (or distillation of models with them...) From my reading of the Fable 5 and Opus 5 System cards, my reconstruction is something like: Finish the task → make externally observable evidence that it is finished → check your own work → fix problems → don't stop prematurely → satisfy the evaluator comprehensively. That is fantastic for SWE benchmark…
For coding, this is very interesting because the same incentives were present for humans before AI. Tech companies which had a culture of rewarding complexity would see huge Pull Requests and a lot of unnecessary complexity. I've worked in companies which would require a thousand lines of code to implement a feature which would require only a hundred or so lines at a different company. The shorter one was more reliable too. Code begets more code. The incentives created by the company culture had a massive impact... And the culture was heavily determined by whether or not the company had a market monopoly. More monopoly power -> more unnecessary complexity (presumably so that employees could achieve better lock-in/job security through the increased need to manage that complexity; in any case, the company could comfortably afford and it did not present an existential risk as it would in a startup environment).
So it's not surprising that the same dynamics are at play with AI. Now, because code is being churned out so rapidly, the effects have become much more obvious (it took me years to figure this out, but now managers can observe this same effect play out in months); many senior engineers and CTOs will echo my point; but I suspect most engineers and crucially, most managers, still don't get it...
Something tells me that the AI companies supplying the models are well aware of the tradeoff. When you can dial up the complexity of the LLM's output by 5% (I.e. 5% more tokens to solve the same problem) and see a 5% immediate increase in your revenue from a large segment of your users, that's a very tempting knob to dial up! Now when you learn that this complexity compounds and next year's revenue will be 10% higher (purely as a result of your users now having to maintain that additional complexity); this is extremely tempting! Especially in the context of users who are largely ignorant as to the true cost of the unnecessary complexity they are adding... The insider's term for this is 'technical debt' for multiple reasons; including the fact that it compounds like normal debt. Now factor in the monopolistic tendencies of those tech markets... It's a real bottomless gold mine.
Revenue from downstream corporate users comes in regardless of compounding code complexity and slower pace of delivery; those companies just keep hiring more people, spending more on tokens. Swallowing up these massive diminishing returns like an appetizer. Enshittification takes place but the downstream end user has nowhere else to go.
It's trivial for a lab to advertise themselves as being token-efficient and almost impossible for its corporate users to actually verify it.
The only real issue with that business model (possibly a fatal issue) are these open weights models which the big tech companies could use to move off the AI service platforms if the problem becomes bad enough.
Re: Qwen 3.8 27B is excellent, but it defaults to overthinking things
#73All current era models overthink as it's a product of their RL incentives (or distillation of models with them...) From my reading of the Fable 5 and Opus 5 System cards, my reconstruction is something like: Finish the task → make externally observable evidence that it is finished → check your own work → fix problems → don't stop prematurely → satisfy the evaluator comprehensively. That is fantastic for SWE benchmark…
I believe this is what Meta is doing as they started recording their SWE screens some time ago.
Re: Qwen 3.8 27B is excellent, but it defaults to overthinking things
#74All current era models overthink as it's a product of their RL incentives (or distillation of models with them...) From my reading of the Fable 5 and Opus 5 System cards, my reconstruction is something like: Finish the task → make externally observable evidence that it is finished → check your own work → fix problems → don't stop prematurely → satisfy the evaluator comprehensively. That is fantastic for SWE benchmark…
The recent 'Stolen Thoughts'[1] paper shows many excerpts of private reasoning for frontier models. For a complex maths problem, Sol reasoned in 367 tokens before working: We need solve. Need interpret no person sits next to two other people = among binary string length16 weight8, no occupied chair whose both neighbors occupied, equivalently ab 111 substring. Endpoints cannot have two neighbors anyway; only avoid 111…
All these weird partial language thought patterns im surprised none of the teams have taught the models to think in something like court stenography or some very dense pattern (i mean they even tried caveman language at one point)
Re: Qwen 3.8 27B is excellent, but it defaults to overthinking things
#75So if I have to set this up on my 24GB MBP what'd the right configuration and tuning look like?
Checkout MTPLX. I use the Youssofal/Qwen3.8-27B-MTPLX-Optimized-Speed model and I get around 12 tok/s on a 36GB MBP.
Re: Qwen 3.8 27B is excellent, but it defaults to overthinking things
#76Ironically I had just installed omlx, tried 3.8 27b 8bit and then Googled about it overthinking, then this was the first result. 4 hours old.
Re: Qwen 3.8 27B is excellent, but it defaults to overthinking things
#77I feel like the current “reasoning” that LLMs are doing has got to be a dead end eventually. Every time I have to read another answer with “but wait” and “Actually,” as they “reason” their way to a (sometimes) better answer, I feel like there’s got to be a way to just shortcut to the actual correct answer instead of burning all these token going in circles mimicking actual thought
I hated it at first too...
Now though I'm considering all the hidden "thinking" in the models layers that happens for each token output. It is a wild amount of waste! We just can't see it.
This kind of stupid excessive computation is fundamentally how these models are so good.
One day hopefully not so soon someone smart or a foundation model will come up with a more efficient architecture. That's when things get really scary.
Re: Qwen 3.8 27B is excellent, but it defaults to overthinking things
#78Ironically I had just installed omlx, tried 3.8 27b 8bit and then Googled about it overthinking, then this was the first result. 4 hours old.
The feedback loops are getting tighter every day.
Re: Qwen 3.8 27B is excellent, but it defaults to overthinking things
#79All current era models overthink as it's a product of their RL incentives (or distillation of models with them...) From my reading of the Fable 5 and Opus 5 System cards, my reconstruction is something like: Finish the task → make externally observable evidence that it is finished → check your own work → fix problems → don't stop prematurely → satisfy the evaluator comprehensively. That is fantastic for SWE benchmark…
The recent 'Stolen Thoughts'[1] paper shows many excerpts of private reasoning for frontier models. For a complex maths problem, Sol reasoned in 367 tokens before working: We need solve. Need interpret no person sits next to two other people = among binary string length16 weight8, no occupied chair whose both neighbors occupied, equivalently ab 111 substring. Endpoints cannot have two neighbors anyway; only avoid 111…
Openai has been focusing a lot on cutting down overthinking is the feel I get. If you look at the artificial analysis tokens per task benchmark Sol especially at lower effort uses far less tokens than the competition.
Re: Qwen 3.8 27B is excellent, but it defaults to overthinking things
#80Earlier quoted context omitted.
Apple is reportedly considering skipping the higher-end M6 chips altogether, which could potentially give enough time for higher-end M7 (in over a year) to use HBM
Considering, or being forced to consider due to the hardware crunch?
Who cares? If high-end M7 chips come out with HBM and actually compete with GPUs from this decade I'll be really happy. I'm also hoping for more than 128GB of unified memory in a laptop.