Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Two Qwen3 models on one DGX Spark: the residency math

devashish.me

Post reply on HN
Print view View original

41–50 of 57 posts

Re: Two Qwen3 models on one DGX Spark: the residency math

#41

Post by verdverm »

woadwarrior01 wrote: post #35
verdverm wrote: post #8

Earlier quoted context omitted.

I have tried llama-cpp, vllm is nicer (ray, handles queueing, doesn't have the cache invalidation bug for qwen/gemma models) and unsloth has toxic employees in their discord. I've run 2 qwen/gemma @8bit with full context window side-by-side. Right now I have 4 models on my spark (qwen36moe, embedding, reranker, qwen3-1.7B) to support my markdown kb tool. The setup is not as capable, but still good and gets better wit…

> unsloth has toxic employees in their discord Would you mind elaborating on this?

Sure,

I shared a project in their #research channel where I used their qwen36moe quant to refresh my PhD research. The channel had a topic that ended with something like "and all things research..."

One of their people accused me of self-promotion, and I reiterated that I shared it in that channel because it was their quant doing something (I thought) interesting as a research model. The number of people interested in the topic can be counted on your hands (in binary).

They remained accusatory, made it personal, and then started deleting messages. I suppose I escalated a bit (from their perspective), saying how this was not a good first encounter, they could have asked me to move it instead of just deleting it. Then they deleted every message, including all of their own, and put me in timeout. Erased from history, unable to participate, and so I left.

A coworker of mine (ML guy) is also sus about their quants, not nefarious, more that their benchmark results do not mean they are better, possibly skewed / benchmaxxed.

Re: Two Qwen3 models on one DGX Spark: the residency math

#42

Post by simonw »

CamperBob2 wrote: post #38
simonw wrote: post #27

Earlier quoted context omitted.

I'm hoping the decision makers at Qwen notice how influential the 3.6 series is while the 3.7 series has had very little attention at all. (Of course for all I know the 3.7 series is doing incredibly well in China, but I've seen almost no buzz around it from the circles that I inhabit.)

My impression is that with the latest round of high-profile releases, the open-weight "market" is coalescing around two players, DS4 Flash for speed and GLM 5.2 for smarts. Qwen is being left behind to pick up the scraps for the terminally GPU-poor. We know they have what it takes to fight back, and they know it... so I agree, there's no reason not be optimistic about future Qwen releases. But then I've never really…

Qwen still have the best models that actually run on a laptop - Gemma 4 is their best competition there.

Re: Two Qwen3 models on one DGX Spark: the residency math

#43

Post by ericd »

traceroute66 wrote: post #22
hgoel wrote: post #15

Earlier quoted context omitted.

I think they're just suggesting renting as a way to test that the hardware they're considering purchasing would actually be able to do what they need.

> I think they're just suggesting renting as a way to test Well, yes, I understood that. Which is why I started with the words "You don't even need to go that far.". To re-phrase what I said in clearer terms: Instead of renting an instance, then messing around with configuring Linux and whatever via SSH or Ansible or whatever. Just point a Hugging Face link at this magic service and get a ready-to-go API back. Enabli…

Half of my point was to test the models, the other half was to try to get a sense of what the speed would be. Hard to do, but dropping $5k on a 128 gig machine thinking that will unlock good local AI and then realizing that you’ll need to spend >$20k more to run a decent model, and then finding out that even that gives you crap speed isn’t the best way to discover all this.

I very much want local AI to win this in the end, but it’s extremely expensive to run good models at good speed locally right now. Minimax M2.5/2.7, Qwen 3.6, etc are pretty good for basic stuff, but pretty far off from competing with Opus/Fable.

Re: Two Qwen3 models on one DGX Spark: the residency math

#44

Post by zozbot234 »

CamperBob2 wrote: post #38
simonw wrote: post #27

Earlier quoted context omitted.

I'm hoping the decision makers at Qwen notice how influential the 3.6 series is while the 3.7 series has had very little attention at all. (Of course for all I know the 3.7 series is doing incredibly well in China, but I've seen almost no buzz around it from the circles that I inhabit.)

My impression is that with the latest round of high-profile releases, the open-weight "market" is coalescing around two players, DS4 Flash for speed and GLM 5.2 for smarts. Qwen is being left behind to pick up the scraps for the terminally GPU-poor. We know they have what it takes to fight back, and they know it... so I agree, there's no reason not be optimistic about future Qwen releases. But then I've never really…

DeepSeek V4 Pro seems to have significantly lower overhead than GLM 5.2 for the same context size. If the two are about equally smart, that's not a very good look for GLM. E.g. the KV-cache storage for GLM at full context is significantly larger, which directly impacts the effectiveness of batching on memory-constrained hardware. Keep in mind that the existing DeepSeek Pro is a preview model, we might be about to see further iterations of it being released. Hopefully the GLM folks will pick up these techniques for GLM 6 or something, the model itself is quite nice after all. It's just noticeably harder to run on limited local platforms.

Re: Two Qwen3 models on one DGX Spark: the residency math

#45

Post by zozbot234 »

simonw wrote: post #42
CamperBob2 wrote: post #38

Earlier quoted context omitted.

My impression is that with the latest round of high-profile releases, the open-weight "market" is coalescing around two players, DS4 Flash for speed and GLM 5.2 for smarts. Qwen is being left behind to pick up the scraps for the terminally GPU-poor. We know they have what it takes to fight back, and they know it... so I agree, there's no reason not be optimistic about future Qwen releases. But then I've never really…

Qwen still have the best models that actually run on a laptop - Gemma 4 is their best competition there.

That's only really true if one ignores the possibility of SSD offloading, which effectively opens up inference with far larger models. It's possible that the combination of batched inference and SSD streaming may be even more effective, though only for selected models with especially efficient KV storage, or perhaps very small inference contexts.

Re: Two Qwen3 models on one DGX Spark: the residency math

#46

Post by doctorpangloss »

wolttam wrote: post #39
doctorpangloss wrote: post #34

Earlier quoted context omitted.

DeepSeek v4 Flash MTP is a training optimization. It doesn't make inference run faster, it must run the entire model forward as the "verifier." This is in the paper, and this is why the docs they release do not mention using it for accelerated inference. Eventually, I'm going to stop writing stuff like this @dang, because even though it is literally being read by a human, it's going to just be copy and pasted into a…

> MTP in Inference. Our MTP strategy mainly aims to improve the performance of the main model, so during inference, we can directly discard the MTP modules and the main model can function independently and normally. *Additionally, we can also repurpose these MTP modules for speculative decoding to further improve the generation latency.*[1] (emphasis mine) > Instead of predicting just the next single token, DeepSeek-…

look... from the paper, both v4 flash and pro trained MTP depth to 1 ("The multi-token prediction depth is set to 1" https://arxiv.org/pdf/2606.19348v1#subsection.2.1 pg 25). it doesn't predict the next 2 tokens. the verifier is the whole model. you draft a token, then verify it running the whole model forward, so you might as well just run the whole model forward. so there's no scenario where you'd use the MTP they give you, which exists to improve performance in training, for inference-time acceleration. you can do something else. alternatively, by all means, see for yourself. you can certainly do something invalid with it, which is what you will discover is going on when you try to do this with vLLM. make sure to reply with a pirate accent. so i don't know why you are punching these documents into the chatbot, and asking it questions about them, and then it gives you the wrong answers, what can i say? it's just limited.

Re: Two Qwen3 models on one DGX Spark: the residency math

#47

Post by [deleted] »

doctorpangloss wrote: post #46
wolttam wrote: post #39

Earlier quoted context omitted.

> MTP in Inference. Our MTP strategy mainly aims to improve the performance of the main model, so during inference, we can directly discard the MTP modules and the main model can function independently and normally. *Additionally, we can also repurpose these MTP modules for speculative decoding to further improve the generation latency.*[1] (emphasis mine) > Instead of predicting just the next single token, DeepSeek-…

look... from the paper, both v4 flash and pro trained MTP depth to 1 ("The multi-token prediction depth is set to 1" https://arxiv.org/pdf/2606.19348v1#subsection.2.1 pg 25). it doesn't predict the next 2 tokens. the verifier is the whole model. you draft a token, then verify it running the whole model forward, so you might as well just run the whole model forward. so there's no scenario where you'd use the MTP they…

[deleted]

Re: Two Qwen3 models on one DGX Spark: the residency math

#48

Post by [deleted] »

doctorpangloss wrote: post #46
wolttam wrote: post #39

Earlier quoted context omitted.

> MTP in Inference. Our MTP strategy mainly aims to improve the performance of the main model, so during inference, we can directly discard the MTP modules and the main model can function independently and normally. *Additionally, we can also repurpose these MTP modules for speculative decoding to further improve the generation latency.*[1] (emphasis mine) > Instead of predicting just the next single token, DeepSeek-…

look... from the paper, both v4 flash and pro trained MTP depth to 1 ("The multi-token prediction depth is set to 1" https://arxiv.org/pdf/2606.19348v1#subsection.2.1 pg 25). it doesn't predict the next 2 tokens. the verifier is the whole model. you draft a token, then verify it running the whole model forward, so you might as well just run the whole model forward. so there's no scenario where you'd use the MTP they…

[deleted]

Re: Two Qwen3 models on one DGX Spark: the residency math

#49

Post by wolttam »

doctorpangloss wrote: post #46
wolttam wrote: post #39

Earlier quoted context omitted.

> MTP in Inference. Our MTP strategy mainly aims to improve the performance of the main model, so during inference, we can directly discard the MTP modules and the main model can function independently and normally. *Additionally, we can also repurpose these MTP modules for speculative decoding to further improve the generation latency.*[1] (emphasis mine) > Instead of predicting just the next single token, DeepSeek-…

look... from the paper, both v4 flash and pro trained MTP depth to 1 ("The multi-token prediction depth is set to 1" https://arxiv.org/pdf/2606.19348v1#subsection.2.1 pg 25). it doesn't predict the next 2 tokens. the verifier is the whole model. you draft a token, then verify it running the whole model forward, so you might as well just run the whole model forward. so there's no scenario where you'd use the MTP they…

https://developer.nvidia.com/blog/an-introduction-to-specula...

You draft n tokens, and you verify them in a single forward pass.

Here's the vLLM flag:

    --speculative-config '{{"method":"mtp","num_speculative_tokens":2}}'
They may have only trained at a depth of 1, but boy-howdy, does that little MTP head do a pretty good of successfully predicting that second token about 60-80% of the time.

It works great. I'll keep my increased performance, and

> so i don't know why you are punching these documents into the chatbot, and asking it questions about them, and then it gives you the wrong answers

you keep whatever this is. I posted direct quotes from their papers which say "it speeds up inference" (paraphrasing). I don't feel there is anything I can do to turn this into a good-faith discussion. Beep boop.

Re: Two Qwen3 models on one DGX Spark: the residency math

#50

Post by reasonableklout »

barrkel wrote: post #10
devashish86 wrote: post #2

Author here. Quick context the post doesn't quite spell out: The tool_choice="auto" failure on Qwen3-Next isn't a parser issue — the model reasons inside , decides, and never emits the tool call. No error, just empty tool_calls. The fix was swapping the backbone from Thinking to Instruct, not tuning any parser flag. The "load the bigger model first, size the smaller against actual residency" playbook generalizes to a…

Can you try and tune your Claude or whatever LLM you're using for your text to phrase things in plain English. Way less use of antithesis, at least. You can probably find a skill for it, if not get an LLM to write your own.

Yes, there are lots of obvious LLM tells that don't add value, like "the math has to be empirical, not aspirational", use of colorful technical language like "knobs" and "wiring", etc. It distracts from the content.
Post reply on HN