Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Two Qwen3 models on one DGX Spark: the residency math

devashish.me

Post reply on HN
Print view View original

31–40 of 57 posts

Re: Two Qwen3 models on one DGX Spark: the residency math

#31

Post by ttsiodras »

binyu wrote: post #16
wolttam wrote: post #13

I started with antirez' DwarfStar[1] on one spark and that (~11-14tok/s generation, ~300-400 tok/s prompt processing) was enough of a taste for me to jump into 2 sparks, running the native quant of DSv4 Flash. Now at 40-50tok/s generation and ~2000 tok/s prefill with a model that I've seen reason through race conditions and be able to trivially pull off any straight-forward coding task, and remain coherent at 500k co…

> Now at 40-50tok/s generation and ~2000 tok/s Not clear how you went from ~11-14 to ~40-50 tok/s. Is it by running the quant native model and adding a second Spark? Cheers

Just chiming in - the claims above are real, I have very similar numbers in a cluster of 2x GX10 I have access to.

Instructions to reproduce, and benchmarks here: https://forums.developer.nvidia.com/t/deepseek-v4-flash-offi...

Re: Two Qwen3 models on one DGX Spark: the residency math

#32

Post by cpburns2009 »

shireboy wrote: post #3

I’ve been considering a move to local llm setup, having been underwhelmed coat vs value of various online offerings. But at the same time worried anything I get will be obsolete in a couple months. And I don’t want to have to babysit it. I really want some agents managing and creating side hustles for me and have some other things. I’m technical-have written my own harness and use gh copilot and grok daily and have a…

Mac, DGX Spark, and a Framework Desktop / Ryzen AI Max 395 (ie Strix Halo) will not give you great performance running LLMs. One benefit of the Spark over the others is you can easily link up to 4 of them. Only MoE (sparse) models will be usable. Even if you can run some massive models, they will crawl. You're better off running one or more GPU cards.

Re: Two Qwen3 models on one DGX Spark: the residency math

#34

Post by doctorpangloss »

wolttam wrote: post #17
binyu wrote: post #16

Earlier quoted context omitted.

> Now at 40-50tok/s generation and ~2000 tok/s Not clear how you went from ~11-14 to ~40-50 tok/s. Is it by running the quant native model and adding a second Spark? Cheers

I suspect DwarfStar could probably squeeze more performance out of the single spark, maybe up closer to 20tok/s. Moving to 2 sparks meant switching to vLLM with 2-way tensor parallelism and working multi-token prediction. The parallelism and MTP on top of better tuned kernels[1] gave an extremely nice boost! I was quite pleased. I've seen bursts up to 60tok/s at ~150k context - sometimes the MTP seems to really kick…

DeepSeek v4 Flash MTP is a training optimization. It doesn't make inference run faster, it must run the entire model forward as the "verifier." This is in the paper, and this is why the docs they release do not mention using it for accelerated inference.

Eventually, I'm going to stop writing stuff like this @dang, because even though it is literally being read by a human, it's going to just be copy and pasted into a chatbot, which will actually spend the time trying to comprehend what I am saying.

Re: Two Qwen3 models on one DGX Spark: the residency math

#35

Post by woadwarrior01 »

verdverm wrote: post #8
dzink wrote: post #4

Have you tried llama.cpp with unsloth and models suited to it? GLM flash? It seemed to allow more models to be tried soon after they are released. Haven’t tried for long term deployment though, that’s the next step.

I have tried llama-cpp, vllm is nicer (ray, handles queueing, doesn't have the cache invalidation bug for qwen/gemma models) and unsloth has toxic employees in their discord. I've run 2 qwen/gemma @8bit with full context window side-by-side. Right now I have 4 models on my spark (qwen36moe, embedding, reranker, qwen3-1.7B) to support my markdown kb tool. The setup is not as capable, but still good and gets better wit…

> unsloth has toxic employees in their discord

Would you mind elaborating on this?

Re: Two Qwen3 models on one DGX Spark: the residency math

#38

Post by CamperBob2 »

simonw wrote: post #27
syhol wrote: post #12

Earlier quoted context omitted.

> So far Someone's optimistic

I'm hoping the decision makers at Qwen notice how influential the 3.6 series is while the 3.7 series has had very little attention at all. (Of course for all I know the 3.7 series is doing incredibly well in China, but I've seen almost no buzz around it from the circles that I inhabit.)

My impression is that with the latest round of high-profile releases, the open-weight "market" is coalescing around two players, DS4 Flash for speed and GLM 5.2 for smarts. Qwen is being left behind to pick up the scraps for the terminally GPU-poor.

We know they have what it takes to fight back, and they know it... so I agree, there's no reason not be optimistic about future Qwen releases. But then I've never really understood what motivates these releases in the first place.

Re: Two Qwen3 models on one DGX Spark: the residency math

#39

Post by wolttam »

doctorpangloss wrote: post #34
wolttam wrote: post #17

Earlier quoted context omitted.

I suspect DwarfStar could probably squeeze more performance out of the single spark, maybe up closer to 20tok/s. Moving to 2 sparks meant switching to vLLM with 2-way tensor parallelism and working multi-token prediction. The parallelism and MTP on top of better tuned kernels[1] gave an extremely nice boost! I was quite pleased. I've seen bursts up to 60tok/s at ~150k context - sometimes the MTP seems to really kick…

DeepSeek v4 Flash MTP is a training optimization. It doesn't make inference run faster, it must run the entire model forward as the "verifier." This is in the paper, and this is why the docs they release do not mention using it for accelerated inference. Eventually, I'm going to stop writing stuff like this @dang, because even though it is literally being read by a human, it's going to just be copy and pasted into a…

> MTP in Inference. Our MTP strategy mainly aims to improve the performance of the main model, so during inference, we can directly discard the MTP modules and the main model can function independently and normally. *Additionally, we can also repurpose these MTP modules for speculative decoding to further improve the generation latency.*[1]

(emphasis mine)

> Instead of predicting just the next single token, DeepSeek-V3 predicts the next 2 tokens through the MTP technique. Combined with the framework of speculative decoding (Leviathan et al., 2023; Xia et al., 2023), it can significantly accelerate the decoding speed of the model.[2]

> As DeepSeek-V3, DeepSeek-V4 series also set MTP modules and objectives. Given that the MTP strategy has been validated in DeepSeek-V3, we adopt the same strategy for DeepSeek-V4 series without modification.[3]

[1]: https://arxiv.org/pdf/2412.19437#subsection.2.2

[2]: https://arxiv.org/pdf/2412.19437#subsubsection.5.4.3

[3]: https://arxiv.org/pdf/2606.19348v1#subsection.2.1

Side comment: I feel you may be too cynical towards your fellow commenters.

Re: Two Qwen3 models on one DGX Spark: the residency math

#40

Post by dofm »

edg5000 wrote: post #26
devashish86 wrote: post #2

Author here. Quick context the post doesn't quite spell out: The tool_choice="auto" failure on Qwen3-Next isn't a parser issue — the model reasons inside , decides, and never emits the tool call. No error, just empty tool_calls. The fix was swapping the backbone from Thinking to Instruct, not tuning any parser flag. The "load the bigger model first, size the smaller against actual residency" playbook generalizes to a…

From the Codex system prompt (verbatim): ``` (...) - Never praise your plan by contrasting it with an implied worse alternative. For example, never use platitudes like \"I will do rather than \", \"I will do , not \". - Never talk about goblins, gremlins, raccoons, trolls, ogres, pigeons, or other animals or creatures unless it is absolutely and unambiguously relevant to the user's query. (...) ``` It seems the OpenA…

FWIW the mere fact that the goblin stuff is necessary and is in the system prompt suggests to me that OpenAI's approach of training ridiculously large models that can do everything for everyone is hopelessly cooked.

Though I concede it is not that much different than straightening the tie of your most valuable employee before you unwisely put them in front of a client and saying "please don't tell them about the regressions they didn't notice and remember, they don't want things explained in allegories drawn from the Silmarillion".

Post reply on HN