Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Two Qwen3 models on one DGX Spark: the residency math

devashish.me

Post reply on HN
Print view View original

21–30 of 57 posts

Re: Two Qwen3 models on one DGX Spark: the residency math

#21

Post by zackify »

ericd wrote: post #6
shireboy wrote: post #3

I’ve been considering a move to local llm setup, having been underwhelmed coat vs value of various online offerings. But at the same time worried anything I get will be obsolete in a couple months. And I don’t want to have to babysit it. I really want some agents managing and creating side hustles for me and have some other things. I’m technical-have written my own harness and use gh copilot and grok daily and have a…

You probably want to try renting some time on a dedicated box with roughly the specs you’re considering and running the open models for a bit to see if you would actually use them before dropping a lot on local hardware. A 128 gig MacBook Pro isn’t going to get you an amazing model, and certainly not amazing speed. GLM 5.2 wants something like 350+ gigs at fp4 iirc.

I ran glm 5.2 on rented 8x h200 it could only do 2x concurrency at a cost of $40 an hour. It felt great but dang I wish it was cheaper... It needs 750 at fp8

Re: Two Qwen3 models on one DGX Spark: the residency math

#22

Post by traceroute66 »

hgoel wrote: post #15
traceroute66 wrote: post #14

Earlier quoted context omitted.

> You probably want to try renting some time on a dedicated box with roughly the specs you’re considering and running the open models You don't even need to go that far. For example, with Exoscale Dedicated Inference[1] you just point it at the Hugging Face for the model and quantisation you want to test and it automagically spits out an OpenAI-compatible API endpoint. [1] https://www.exoscale.com/ai-cloud-infrastruc…

I think they're just suggesting renting as a way to test that the hardware they're considering purchasing would actually be able to do what they need.

> I think they're just suggesting renting as a way to test

Well, yes, I understood that.

Which is why I started with the words "You don't even need to go that far.".

To re-phrase what I said in clearer terms:

Instead of renting an instance, then messing around with configuring Linux and whatever via SSH or Ansible or whatever. Just point a Hugging Face link at this magic service and get a ready-to-go API back. Enabling you to test your desired model spec with minimum fuss.

Ultimately the guy wants his own hardware. So why waste time messing around with someone else's VM if you just want to test a specific model spec. That is the TL;DR.

Re: Two Qwen3 models on one DGX Spark: the residency math

#24

Post by embedding-shape »

pet_the_bird wrote: post #7
dzink wrote: post #4

Have you tried llama.cpp with unsloth and models suited to it? GLM flash? It seemed to allow more models to be tried soon after they are released. Haven’t tried for long term deployment though, that’s the next step.

Highy anecdotal: I have tried various self-hosted models using both vllm and llama.cpp. I am in a situation where I have access to large amount of memory (~320 GB). While experimenting with quantization I found that there is a non-trivial tradeoff between quality and memory footprint. Overall my experience follows the reported pattern of "2-bit is mwah, 4-bit half decent and 6-bit required for programming. Still, alt…

I only have access to 96GB VRAM locally, but I'd agree with the general approach of avoiding lower quantizations, often anything below Q8 seems to suffer greatly on quality and seemingly never worth going below it, better to go for smaller model in that case.

With the exception of DwarfStar + DS4-Flash with IQ2_XXS quantization, which somehow seems to not suffer as much as I'd thought. I'd still opt for a smaller model + at least Q8.

Re: Two Qwen3 models on one DGX Spark: the residency math

#26

Post by edg5000 »

devashish86 wrote: post #2

Author here. Quick context the post doesn't quite spell out: The tool_choice="auto" failure on Qwen3-Next isn't a parser issue — the model reasons inside , decides, and never emits the tool call. No error, just empty tool_calls. The fix was swapping the backbone from Thinking to Instruct, not tuning any parser flag. The "load the bigger model first, size the smaller against actual residency" playbook generalizes to a…

From the Codex system prompt (verbatim):

```

(...) - Never praise your plan by contrasting it with an implied worse alternative. For example, never use platitudes like \"I will do rather than \", \"I will do , not \".

- Never talk about goblins, gremlins, raccoons, trolls, ogres, pigeons, or other animals or creatures unless it is absolutely and unambiguously relevant to the user's query. (...)

```

It seems the OpenAI people added that first bullet to specifically address the tendency the model has, as seen in the parent comment. The goblin stuff coincidentally appears right after in the system prompt, so in included it as a bonus.

Re: Two Qwen3 models on one DGX Spark: the residency math

#27

Post by simonw »

syhol wrote: post #12
simonw wrote: post #11

Earlier quoted context omitted.

So far there aren't any open weight model releases for the Qwen 3.7 family.

> So far Someone's optimistic

I'm hoping the decision makers at Qwen notice how influential the 3.6 series is while the 3.7 series has had very little attention at all.

(Of course for all I know the 3.7 series is doing incredibly well in China, but I've seen almost no buzz around it from the circles that I inhabit.)

Re: Two Qwen3 models on one DGX Spark: the residency math

#30

Post by zackangelo »

zackify wrote: post #21
ericd wrote: post #6

Earlier quoted context omitted.

You probably want to try renting some time on a dedicated box with roughly the specs you’re considering and running the open models for a bit to see if you would actually use them before dropping a lot on local hardware. A 128 gig MacBook Pro isn’t going to get you an amazing model, and certainly not amazing speed. GLM 5.2 wants something like 350+ gigs at fp4 iirc.

I ran glm 5.2 on rented 8x h200 it could only do 2x concurrency at a cost of $40 an hour. It felt great but dang I wish it was cheaper... It needs 750 at fp8

what was the concurrency limitation? that node should be able to support a lot more
Post reply on HN