If anyone else is running this on an RTX 5090, https://github.com/Neroued/ninfer as inference engine gets me ~138 tokens/second, roughly double what I get with a naive llama.cpp setup.
Qwen 3.8 27B
571–580 of 848 posts
Re: Qwen 3.8 27B
#572Earlier quoted context omitted.
What kind of things you only get with a larger model?
Asking it factual information[1]. You just can't compress the entire human knowledge into a 30GB file. [1] Without searching the internet. And even if you allow it, you'll get much worse results because search means browsing and parsing the top results, and search results are horrible, whereas internal knowledge from training encompasses the entire internet plus all books including very niche stuff.
> You just can't compress the entire human knowledge into a 30GB file.
...I'm just asking questions here... how sure are we of this?
If you'd asked me six years ago whether we could compress all of human knowledge into a 1 TB file, I would have said no, and yet here we are.
If you can do 1 TB, why not 30 GB? It's, like, within one order of magnitude.
Re: Qwen 3.8 27B
#573Earlier quoted context omitted.
The people OP mentioned about "just want AI" . The pain point they raised is this is too complicated for people who just want to get started, that is not true anymore. It is certainly fun to fine-tune and setup if you like do something like that, however the need to do it hardly is a barrier for those who don't want complexity as OP imagines. Lower level API/interfaces should not be a barrier for people if they are a…
> More and more people are thinking agent native so this is not really a issue. Why this headache inducing lingo tho? What does that even mean, and why should I sign up for your webinar about that?
The alternative is to label it AI. That is distasteful for some of us, Intelligence is much stronger than automation in our minds so we avoid the term and try to describe it differently .
Re: Qwen 3.8 27B
#574Absolutely the best pelican I've seen from a model that runs on my laptop: https://tools.simonwillison.net/markdown-svg-renderer#url=ht... Bicycle is the right shape. Pelican beak is excellent. Nice background. Most importantly, the pelican has one leg on each side of the bicycle - that's very rare. (No chain on this bicycle though - in the reasoning trace it says "already chainstay... skip chain detail; maybe a smal…
what type of laptop runs something like that? using mac or egpu?
Re: Qwen 3.8 27B
#575The unsloth Q8kxl https://huggingface.co/unsloth/Qwen3.8-27B-GGUF for some reason is looping and going crazy on the think part (I tried to search for an email to let the guys know but didn't find one)... I used the bartowski one and that one doesn't have that issue https://huggingface.co/bartowski/Qwen3.8-27B-GGUF?show_file_... that's using llama.cpp llama-server \ -m ~/somePath/Qwen3.8-27B-UD-Q8_K_XL.gguf \ -np 1 --…
I'm not seeing that with that exact quantization from Unsloth, so far. I'm seeing a _lot_ of thinking before it starts doing, but it all seems pretty reasonable and not loopy (at least no more loopy than big models, with the expected "But, wait! I need to..." types of back-tracking). So, it's taking a long time, but I don't think it's doing anything pathological.
I would characterise it as obsessive, not loopy. It's definitely burning through a lot of tokens to ruminate about aspects of tasks that earlier models get done better seemingly on memory.
I still need to understand that.
Setting a reasoning limit does not seem to have good results, because it really seems to go down rabbit holes and that means that cutting reasoning off too early is going to punish the quality on anything it has not got round to pondering yet. But maybe I have to give it a bit more room.
I have not tested in an agentic sense yet, just with my sort of pet queries in LM Studio, but it rather looks like it expects an agentic flow, because telling it that it's a helpful coding agent and changing the order of things in my prompts (telling it up front to ask any clarifying questions before detailing the rest of the prompt) has definitely kept its thinking a bit more on track.
Re: Qwen 3.8 27B
#576There's a real change (compared to 3.6) in the way it writes in thinking — it drops words like "to" and "we" in "We need to", talks generally in note form, drops the/and all over the place, avoids "for". "Need be helpful concise", "Need maybe not overdo", "Need ask!" Almost caveman. I have an (unsourced, vague) suspicion that this rather unique thinking trace pattern is actually hobbling the MTP predictions, which se…
Annnnd the code of my WP code test is not better. It is bushy, overcomplicated, and has gone around the houses to do stuff it would not need to do if it hadn't overthought. Oh dear. I need to try to understand what is going on here.
thinking is set to max by default. I bet that turning it down would solve this.
Re: Qwen 3.8 27B
#577The unsloth Q8kxl https://huggingface.co/unsloth/Qwen3.8-27B-GGUF for some reason is looping and going crazy on the think part (I tried to search for an email to let the guys know but didn't find one)... I used the bartowski one and that one doesn't have that issue https://huggingface.co/bartowski/Qwen3.8-27B-GGUF?show_file_... that's using llama.cpp llama-server \ -m ~/somePath/Qwen3.8-27B-UD-Q8_K_XL.gguf \ -np 1 --…
I'm not seeing that with that exact quantization from Unsloth, so far. I'm seeing a _lot_ of thinking before it starts doing, but it all seems pretty reasonable and not loopy (at least no more loopy than big models, with the expected "But, wait! I need to..." types of back-tracking). So, it's taking a long time, but I don't think it's doing anything pathological.
Thinking is turned up to max by default. You can turn it down. Unsure why they did this.
Re: Qwen 3.8 27B
#578Absolutely the best pelican I've seen from a model that runs on my laptop: https://tools.simonwillison.net/markdown-svg-renderer#url=ht... Bicycle is the right shape. Pelican beak is excellent. Nice background. Most importantly, the pelican has one leg on each side of the bicycle - that's very rare. (No chain on this bicycle though - in the reasoning trace it says "already chainstay... skip chain detail; maybe a smal…
I was just updating a Proxmox cluster from v8 to v9 and had time on my hands and 15-30 mins after the models appeared for general download I had an initial pelican effort in .svg format.
I switch off reasoning by default and for starters I ran 3.8 the same way as 3.6 on vllm on a DGX Spark box. Open web UI prompted.
My word! It looks rather decent. Much better than previous efforts.
Re: Qwen 3.8 27B
#579If anyone else is running this on an RTX 5090, https://github.com/Neroued/ninfer as inference engine gets me ~138 tokens/second, roughly double what I get with a naive llama.cpp setup.
Amazingly enough, Hacker News decided to show this to me as the top comment, I'm also running an rtx 5090 and trying it out now, thanks for the tip!
Edit: Absolutely blazing fast! Getting 163 tokens/sec on WSL and it generated a pretty sweet Pelican.
https://gist.github.com/hansale/ed9e73fe35165a58ea2af6b1632a...
Re: Qwen 3.8 27B
#580Earlier quoted context omitted.
Annnnd the code of my WP code test is not better. It is bushy, overcomplicated, and has gone around the houses to do stuff it would not need to do if it hadn't overthought. Oh dear. I need to try to understand what is going on here.
thinking is set to max by default. I bet that turning it down would solve this.
I think so too — it is something to test, for sure.
ETA: a bit of testing before I climb the wooden hill to Bedfordshire.
LM Studio doesn't seem to display the little dropdown to set reasoning effort, so I bodged the chat template on load to get it to choose 'medium'.
As soon as you switch away from xhigh, it goes back to thinking in normal sentences like Qwen 3.6, rather than in sort of quasi caveman.
And you get all the Wait, Actually, No wait… stuff back.
And it is behaving a lot more like it used to. So that is pretty interesting.