How about Qwen3.6? What sort of prefill/decode rates? Edit: 3.6 not 3.7!
So far there aren't any open weight model releases for the Qwen 3.7 family.
Two Qwen3 models on one DGX Spark: the residency math
11–20 of 57 posts
Re: Two Qwen3 models on one DGX Spark: the residency math
#12Re: Two Qwen3 models on one DGX Spark: the residency math
#13I started with antirez' DwarfStar[1] on one spark and that (~11-14tok/s generation, ~300-400 tok/s prompt processing) was enough of a taste for me to jump into 2 sparks, running the native quant of DSv4 Flash.
Now at 40-50tok/s generation and ~2000 tok/s prefill with a model that I've seen reason through race conditions and be able to trivially pull off any straight-forward coding task, and remain coherent at 500k context. With a preview checkpoint of the weights!
I'm excited for the future of local LLMs. There is some buy-in but apparently not an extreme amount to get access to models that can stand in the for the giants on all but the most challenging and/or hands-off coding tasks.
Re: Two Qwen3 models on one DGX Spark: the residency math
#14I’ve been considering a move to local llm setup, having been underwhelmed coat vs value of various online offerings. But at the same time worried anything I get will be obsolete in a couple months. And I don’t want to have to babysit it. I really want some agents managing and creating side hustles for me and have some other things. I’m technical-have written my own harness and use gh copilot and grok daily and have a…
You probably want to try renting some time on a dedicated box with roughly the specs you’re considering and running the open models for a bit to see if you would actually use them before dropping a lot on local hardware. A 128 gig MacBook Pro isn’t going to get you an amazing model, and certainly not amazing speed. GLM 5.2 wants something like 350+ gigs at fp4 iirc.
> You probably want to try renting some time on a dedicated box with roughly the specs you’re considering and running the open models
You don't even need to go that far. For example, with Exoscale Dedicated Inference[1] you just point it at the Hugging Face for the model and quantisation you want to test and it automagically spits out an OpenAI-compatible API endpoint.
[1] https://www.exoscale.com/ai-cloud-infrastructure/dedicated-i...
(I have no relationship with Exoscale, this particular product just crossed my radar recently)
Re: Two Qwen3 models on one DGX Spark: the residency math
#15Earlier quoted context omitted.
You probably want to try renting some time on a dedicated box with roughly the specs you’re considering and running the open models for a bit to see if you would actually use them before dropping a lot on local hardware. A 128 gig MacBook Pro isn’t going to get you an amazing model, and certainly not amazing speed. GLM 5.2 wants something like 350+ gigs at fp4 iirc.
> You probably want to try renting some time on a dedicated box with roughly the specs you’re considering and running the open models You don't even need to go that far. For example, with Exoscale Dedicated Inference[1] you just point it at the Hugging Face for the model and quantisation you want to test and it automagically spits out an OpenAI-compatible API endpoint. [1] https://www.exoscale.com/ai-cloud-infrastruc…
I think they're just suggesting renting as a way to test that the hardware they're considering purchasing would actually be able to do what they need.
Re: Two Qwen3 models on one DGX Spark: the residency math
#16I started with antirez' DwarfStar[1] on one spark and that (~11-14tok/s generation, ~300-400 tok/s prompt processing) was enough of a taste for me to jump into 2 sparks, running the native quant of DSv4 Flash. Now at 40-50tok/s generation and ~2000 tok/s prefill with a model that I've seen reason through race conditions and be able to trivially pull off any straight-forward coding task, and remain coherent at 500k co…
> Now at 40-50tok/s generation and ~2000 tok/s
Not clear how you went from ~11-14 to ~40-50 tok/s. Is it by running the quant native model and adding a second Spark?
Cheers
Re: Two Qwen3 models on one DGX Spark: the residency math
#17I started with antirez' DwarfStar[1] on one spark and that (~11-14tok/s generation, ~300-400 tok/s prompt processing) was enough of a taste for me to jump into 2 sparks, running the native quant of DSv4 Flash. Now at 40-50tok/s generation and ~2000 tok/s prefill with a model that I've seen reason through race conditions and be able to trivially pull off any straight-forward coding task, and remain coherent at 500k co…
> Now at 40-50tok/s generation and ~2000 tok/s Not clear how you went from ~11-14 to ~40-50 tok/s. Is it by running the quant native model and adding a second Spark? Cheers
I suspect DwarfStar could probably squeeze more performance out of the single spark, maybe up closer to 20tok/s.
Moving to 2 sparks meant switching to vLLM with 2-way tensor parallelism and working multi-token prediction. The parallelism and MTP on top of better tuned kernels[1] gave an extremely nice boost! I was quite pleased. I've seen bursts up to 60tok/s at ~150k context - sometimes the MTP seems to really kick in (i.e. high acceptance rate on its tokens)
Currently running a custom vLLM build put together by some folks on the Nvidia forums[2], which speaks to how early support for the model is.
Re: Two Qwen3 models on one DGX Spark: the residency math
#18Earlier quoted context omitted.
> Now at 40-50tok/s generation and ~2000 tok/s Not clear how you went from ~11-14 to ~40-50 tok/s. Is it by running the quant native model and adding a second Spark? Cheers
I suspect DwarfStar could probably squeeze more performance out of the single spark, maybe up closer to 20tok/s. Moving to 2 sparks meant switching to vLLM with 2-way tensor parallelism and working multi-token prediction. The parallelism and MTP on top of better tuned kernels[1] gave an extremely nice boost! I was quite pleased. I've seen bursts up to 60tok/s at ~150k context - sometimes the MTP seems to really kick…
Personally, I've tried to squeeze more tok/s for a single DGX Spark deployment and DeepSeek V4 Flash but only got marginal improvements. There's work to do on fusing kernels and other optimizations that are already on antirez's roadmap so it is not worth duplicating efforts.
I've had positive experiences running GLM 4.7 via vLLM, tool calling works well and the inference is fast. Do you run DeepSeek V4 Flash on vLLM?
Re: Two Qwen3 models on one DGX Spark: the residency math
#19I’ve been considering a move to local llm setup, having been underwhelmed coat vs value of various online offerings. But at the same time worried anything I get will be obsolete in a couple months. And I don’t want to have to babysit it. I really want some agents managing and creating side hustles for me and have some other things. I’m technical-have written my own harness and use gh copilot and grok daily and have a…
Well, if you are making side-hustle money now using online models that, critically, you could also run at home, then it sounds like it’s just a matter of numbers. Oh and, unless you spend a lot more than 5k, your local model will still be slower than the online model. What’s your estimated ROI? Assuming that’s not true based on your phrasing, you’d be shooting yourself in the foot. Start using online models with the…
no one is making money side-hustling ai models. This is like reddit wet dream. get real, dont get scammed by ppl selling you these dreams.
Re: Two Qwen3 models on one DGX Spark: the residency math
#20Earlier quoted context omitted.
I suspect DwarfStar could probably squeeze more performance out of the single spark, maybe up closer to 20tok/s. Moving to 2 sparks meant switching to vLLM with 2-way tensor parallelism and working multi-token prediction. The parallelism and MTP on top of better tuned kernels[1] gave an extremely nice boost! I was quite pleased. I've seen bursts up to 60tok/s at ~150k context - sometimes the MTP seems to really kick…
Personally, I've tried to squeeze more tok/s for a single DGX Spark deployment and DeepSeek V4 Flash but only got marginal improvements. There's work to do on fusing kernels and other optimizations that are already on antirez's roadmap so it is not worth duplicating efforts. I've had positive experiences running GLM 4.7 via vLLM, tool calling works well and the inference is fast. Do you run DeepSeek V4 Flash on vLLM?
Yep, those are the numbers I'm getting with DSv4 Flash on vLLM across 2 sparks.