Idea: Qwen should change its name to OpenQwen - this would probably 10x their usage. :D
The word "open" has lost all meaning
Qwen 3.8 27B is excellent, but it defaults to overthinking things
221–230 of 411 posts
Re: Qwen 3.8 27B is excellent, but it defaults to overthinking things
#222Earlier quoted context omitted.
Out of the loop here. What did Google and unsloth and llama do to mutilate Gemma? I can understand Google shenanigans but llama and gunsmith is kind of surprising.
Google provided incorrect settings and an imperfect template. Unsloth modified the template and then finetuned their own version of the model to optimize for some benchmarks as a means of validating quants. Google and Llama.cpp then adopt template changes by default, so anyone downloading the new model or even using the original model will now automatically be using it incorrectly. Llama.cpp also uses the same infere…
I am very much a beginner to local LLM stuff and I find it incredibly hard to figure out how to run models optimally with the correct settings for my hardware. The number of different variations of the same model and how each quant work is super confusing as well.
When I tried to run llama.cpp directly I was getting max 9tk/s on qwen3.5-9B, then I tried LM Studio with the same model and got 77tk/s. I haven't figured out yet how to get MTP working properly in either.
Re: Qwen 3.8 27B is excellent, but it defaults to overthinking things
#223Earlier quoted context omitted.
> I keep hearing this claim, and yet I keep seeing LLM output which is trivially distinguished from human writing. That's mostly true for longer LLM output with all the sycophancy / LinkedIn bias thrown in. Make it casual conversation or comments, and give it instructions on appearing casual, or even better kill the censoring and fixed-prompt (with an open model), and it's orders of magnitude more difficult, unless i…
At the bottom of this very submission are a bunch of dead comments that are very obviously LLM-generated.
Ok, you and I can easily spot LLM text. So what? The Turing test has still been passed, as is clear by people falling in love with ChatGPT, not believing something is AI, and by continuously claiming this or that is a bot.
People, many of them at least, cannot make this distinction anymore. You can, I can, but people as a whole are having problems with that.
Re: Qwen 3.8 27B is excellent, but it defaults to overthinking things
#224It won't satisfy the people who just want to drop a model into their existing toolset and run, but I think there are a lot of ways to deal with this overthinking problem. For instance, it's a step backward, but I put {"reasoning_effort":"none"} and led it by the nose: User: We're going to make . Please create a plan, but do not write code yet. Agent: User: Now please follow that plan and write the code. No other chat…
My impression is that when you allow the model to use internal thinking as opposed to asking it to output its thinking first, it's more likely to backtrack when I detects a flaw in its plan. Said otherwise: once producing user-facing output it seems to lock into an approach, for better or worse.
true but thats not how we work. We see a problem, we make a plan and then we adjust the plan as we find the flaws.
trying to reason about all the ways it can go wrong after a point just stops one from starting the task. Which is exactly what I find with models.
Re: Qwen 3.8 27B is excellent, but it defaults to overthinking things
#225Earlier quoted context omitted.
Out of the loop here. What did Google and unsloth and llama do to mutilate Gemma? I can understand Google shenanigans but llama and gunsmith is kind of surprising.
Google provided incorrect settings and an imperfect template. Unsloth modified the template and then finetuned their own version of the model to optimize for some benchmarks as a means of validating quants. Google and Llama.cpp then adopt template changes by default, so anyone downloading the new model or even using the original model will now automatically be using it incorrectly. Llama.cpp also uses the same infere…
[dead]
Re: Qwen 3.8 27B is excellent, but it defaults to overthinking things
#226Earlier quoted context omitted.
Full agree. I until very recently thought AI tools of today were limited to prohibitively expensive high end hardware hosted in data centers. I was surprised and amazed to get "decent" (with the expectations set right / low) coding performance out of Qwen3.5-9B on a decidedly medium end Radeon 9070 paired with a 5700x3d and 32GB of DDR4 RAM. We can finally reason with and "talk" to our hardware.
Yes, and we are still pretty early: AI is still advancing at breakneck speeds, and hardware is too.
Is the hardware really getting better? It feels performance per watt is not getting better at all which is the metric that will matter eventually when supply-demand stabilizes.
As it is, it seems the improvements are about making the hardware cheaper (as in capex, not opex).
This is just feels from me from what I hear on the news and see on the products though.
Re: Qwen 3.8 27B is excellent, but it defaults to overthinking things
#227“The fact that a 17GB file can do all of this stuff on my home machines is a miracle. Once again, I’m delighted and amazed at how much progress local models have made this year.” I think that should be the blinking headline - this shows what can be done with consumer hardware.
For me that moment was Gemma 4 12B QAT. You're not suddenly going to start throwing your hardest programming problems at Gemma 4 12B QAT, it is still 15B parameters less. It's more that, aside from pelican art which isn't what local models are for, I didn't see anything on Simon's post that it couldn't assist with or largely succeed at. It can run 80-100t/s on a laptop, can understand images natively and do bounding…
> It can run 80-100t/s on a laptop
That is a lot, what is your laptop hardware?
One issue I have with Gemma is that they seem to use old architectures that rely on full attention, requiring a lot of RAM for context and quickly degrading speeds as context is filled.
Qwen 3.5+ is much better in that regard with its super efficient context. Even on Macs, speeds take degrade much more slowly.
Re: Qwen 3.8 27B is excellent, but it defaults to overthinking things
#228Earlier quoted context omitted.
Out of the loop here. What did Google and unsloth and llama do to mutilate Gemma? I can understand Google shenanigans but llama and gunsmith is kind of surprising.
Google provided incorrect settings and an imperfect template. Unsloth modified the template and then finetuned their own version of the model to optimize for some benchmarks as a means of validating quants. Google and Llama.cpp then adopt template changes by default, so anyone downloading the new model or even using the original model will now automatically be using it incorrectly. Llama.cpp also uses the same infere…
And what's the right way to use Gemma? Where can I find the correct template and settings if those aren't the ones provided by Google, Unsloth, and aren't built into llama.cpp? I discarded using Gemma 4 because it got into weird loops when tool calling
Re: Qwen 3.8 27B is excellent, but it defaults to overthinking things
#229Earlier quoted context omitted.
> I keep hearing this claim, and yet I keep seeing LLM output which is trivially distinguished from human writing. That's mostly true for longer LLM output with all the sycophancy / LinkedIn bias thrown in. Make it casual conversation or comments, and give it instructions on appearing casual, or even better kill the censoring and fixed-prompt (with an open model), and it's orders of magnitude more difficult, unless i…
At the bottom of this very submission are a bunch of dead comments that are very obviously LLM-generated.
And that is proof that all LLM comments are easily found out? Also, easily found out by average humans? (this is not a average forum here)
Re: Qwen 3.8 27B is excellent, but it defaults to overthinking things
#230Earlier quoted context omitted.
$0.286/kWh is a ridiculous amount of money to pay for power. That's more than double the regional residental average here! If I ever found myself in this situation I would much rather just rent cards from hotasile and run open models instead of giving OAI money and playing reset bingo
I removed the extra links to sources for the electricity prices, but that's the average cost in EU, where I live. https://ec.europa.eu/eurostat/web/products-eurostat-news/w/d...
it is not the marginal price for household users, but the (average electricity bill : subscription + consumption) / (average consumption), so it will be higher than the additional kwh consumed.