Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Nvidia DGX Spark

nvidia.com

Post reply on HN
Print view View original

181–190 of 222 posts

Re: Nvidia DGX Spark

#181

Post by UncleOxidant »

hasperdi wrote: post #97
UncleOxidant wrote: post #90

Earlier quoted context omitted.

> Ryzen AI Max 395+, ~120 tops (fp8?), 128GB RAM, $1999 Just got my Framework PC last week. It's easy to setup to run LLMs locally - you have to use Fedora 42, though, because it has the latest drivers. It was super easy to get qwen3-coder-30b (8 bit quant) running in LMStudio at 36 tok/sec.

Hi could you share if you get a decent coding performance (quality wise) with this setup? IE. Is it good enough to replace say Claude Code?

qwen3-coder-30b is surprisingly good for a smallish model, but it's not going to replace Claude Code. Maybe if you're using it for Python it could do well enough. I've been trying it with C code generation and it's not bad, but certainly not at Claude Code level. I hope they come out with a qwen coder model in the 60b to 80b range - something like that would give higher quality results and likely still run in the 15 tok/sec range which would be usable.

Re: Nvidia DGX Spark

#182

Post by UncleOxidant »

alias_neo wrote: post #143
UncleOxidant wrote: post #90

Earlier quoted context omitted.

> Ryzen AI Max 395+, ~120 tops (fp8?), 128GB RAM, $1999 Just got my Framework PC last week. It's easy to setup to run LLMs locally - you have to use Fedora 42, though, because it has the latest drivers. It was super easy to get qwen3-coder-30b (8 bit quant) running in LMStudio at 36 tok/sec.

I'm pretty new to this, so if I wanted to benchmark my current hardware and compare to your results what would be the best way to do that? I'm looking at going for a Framework Desktop and would like to know what kind of performance gain I'd get over the current hardware I have, which so far I have a "feel" for the performance of from running Ollama and OpenWebUI, but no hard numbers.

You could load up LMStudio on your current hardware, get qwen3-coder-30b (8bit quant) and give it some coding tasks, something meaty (I had it create a recursive descent parser in C that parses the C programming language). At the end of it's response it shows the tok/sec. I'm getting 36 tok/sec on the Framework running that model.

Re: Nvidia DGX Spark

#183

Post by UncleOxidant »

pixelpoet wrote: post #118
UncleOxidant wrote: post #90

Earlier quoted context omitted.

> Ryzen AI Max 395+, ~120 tops (fp8?), 128GB RAM, $1999 Just got my Framework PC last week. It's easy to setup to run LLMs locally - you have to use Fedora 42, though, because it has the latest drivers. It was super easy to get qwen3-coder-30b (8 bit quant) running in LMStudio at 36 tok/sec.

Very encouraging result, I'm waiting super anxiously for mine! How much memory did you allocate for the iGPU?

I haven't done any fiddling with that yet. Out of the box it seems to allocate 1/2 for the iGPU. The qwen3-coder-30b 8bit quant model was (as you would expect) only taking 30GB (a bit less than half of what was allocated). Though weirdly, in htop it shows that the CPU has 125GB available to it, so I'm not sure what to make of that.

Re: Nvidia DGX Spark

#184

Post by robbomacrae »

cherioo wrote: post #24

The mainstream options seem to be Ryzen AI Max 395+, ~120 tops (fp8?), 128GB RAM, $1999 Nvidia DGX Spark, ~1000 tops fp4, 128GB RAM, $3999 Mac Studio max spec, ~120 tflops (fp16?), 512GB RAM, 3x bandwidth, $9499 DGX Spark appears to potentially offer the most token per second, but less useful/value as everyday pc.

GosuCoder's latest video seems to be a well timed test of using Ryzen AI Max on some local models getting 40 TPS on a quantized Qwen 3 coder.

https://www.youtube.com/watch?v=0DET4YFzS6A

Re: Nvidia DGX Spark

#185

Post by jweir »

spwa4 wrote: post #137
KingOfCoders wrote: post #87

Earlier quoted context omitted.

In Germany the 96gb version is 5000 EUR and the 256gb version is 7000 EUR (no 128gb available as far as I can see).

At that point it's far superior to fly to the US, buy it, and fly back. Hell, have a nice week in a hotel and bring two.

Fly to a state with no sales tax. Portland, Oregon serves this purpose for high end shoppers that come from out of state and out of the country. Folks fly in to buy their Rolex, Gucci, etc, with no tax.

Re: Nvidia DGX Spark

#186

Post by thomasskis »

aurareturn wrote: post #132
thomasskis wrote: post #105

Earlier quoted context omitted.

I run 4 Mac Studio ultras at work (they’re pricy when maxed out), for local-first AI dev services. But there’s a few things that make me want to switch to the Spark. Networking is the biggest one, the Macs have Thunderbolt and Ethernet, but if I run distributed inference with EXO over Thunderbolt; the drop in tokens/second is massive. These Sparks get RDMA and can stack nicely. The other big one is access to CUDA, ML…

You'd have to stack 16 of these to get 2TB of VRAM, equivalent to 4 Mac Studios 512GBs chained together. 16 compared to 4. Surely even much faster networking in the Spark would degrade with that many devices? Biggest problem with Macs is that they don't have dedicated tensor cores in the GPU which makes prompt processing very slow compared to Nvidia and AMD.

It’s $12k for each Mac Studio, and the networking makes them only effective individually (it’s like less that 15 tokens/s with EXO) while NVLINK is very effective. The Spark is definitely more scalable, but the MLX and metal teams are cooking, so honestly either way is still winning.

Re: Nvidia DGX Spark

#187

Post by hnuser123456 »

ilirium wrote: post #160
artemisart wrote: post #149

Earlier quoted context omitted.

Ok then just to clarify: you can fit 4x larger models on the Spark vs 5090, not 17x.

@nabla9 have tried to tell you that for DGX Spark, you can also use optimized models; therefore, this means that Spark can also be used for inference with bigger models, such as those exceeding 200B. Please compare the same things: carrots VS carrots, not apples VS eggs.

You and nabla9 are both the one comparing apples and eggs. 4x more RAM means 4x larger models when everything else is held the same to make a fair comparison.

Re: Nvidia DGX Spark

#188

Post by hnuser123456 »

bjackman wrote: post #119
hereme888 wrote: post #49

FP4-sparse (TFLOPS) | Price | $/TF4s 5090: 3352 | 1999 | 0.60 Thor: 2070 | 3499 | 1.69 Spark: 1000 | 3999 | 4.00 ____________ FP8-dense (TFLOPS) | Price | $/TF8d (4090s have no FP4) 4090 : 661 | 1599 | 2.42 4090 Laptop: 343 | vary | - ____________ Geekbench 6 (compute score) | Price | $/100k 4090: 317800 | 1599 | 503 5090: 387800 | 1999 | 516 M4 Max: 180700 | 1999 | 1106 M3 Ultra: 259700 | 3999 | 1540 ____________ Ap…

Note you cannot actually get a 5090 for $1999 that's just the RRP. I believe they actually cost $4k

The prices came down to near MSRP in the last month or so.

Re: Nvidia DGX Spark

#189

Post by linuxftw »

alias_neo wrote: post #143
UncleOxidant wrote: post #90

Earlier quoted context omitted.

> Ryzen AI Max 395+, ~120 tops (fp8?), 128GB RAM, $1999 Just got my Framework PC last week. It's easy to setup to run LLMs locally - you have to use Fedora 42, though, because it has the latest drivers. It was super easy to get qwen3-coder-30b (8 bit quant) running in LMStudio at 36 tok/sec.

I'm pretty new to this, so if I wanted to benchmark my current hardware and compare to your results what would be the best way to do that? I'm looking at going for a Framework Desktop and would like to know what kind of performance gain I'd get over the current hardware I have, which so far I have a "feel" for the performance of from running Ollama and OpenWebUI, but no hard numbers.

What nobody seems to ever share is the context and TTFT (time to first token). You can get a very good TPS by using small prompts, even if the output tokens are very large. If you try to do any kind of agentic coding locally, where contexts are 7k+, local hardware completely falls over.

qwen-code (cli) gives like 2k requests per day for free (and is fantastic), so unless you have a very specific use case, buying a system for local LLM use is not a good use of funds.

If you're in the market for a desktop PC anyway, and just want to tinker with LLMs, then the AMD systems are a fair value IMO, plus the drivers are open source so everything just works out of the box (with Vulkan, anyway).

Re: Nvidia DGX Spark

#190

Post by UncleOxidant »

linuxftw wrote: post #189
alias_neo wrote: post #143

Earlier quoted context omitted.

I'm pretty new to this, so if I wanted to benchmark my current hardware and compare to your results what would be the best way to do that? I'm looking at going for a Framework Desktop and would like to know what kind of performance gain I'd get over the current hardware I have, which so far I have a "feel" for the performance of from running Ollama and OpenWebUI, but no hard numbers.

What nobody seems to ever share is the context and TTFT (time to first token). You can get a very good TPS by using small prompts, even if the output tokens are very large. If you try to do any kind of agentic coding locally, where contexts are 7k+, local hardware completely falls over. qwen-code (cli) gives like 2k requests per day for free (and is fantastic), so unless you have a very specific use case, buying a sy…

> If you're in the market for a desktop PC anyway, and just want to tinker with LLMs, then the AMD systems are a fair value

Yeah, this is why I bought it. To tinker with LLMs (and some more experimental ML algorithms like differential logic and bitnets), but also it can compile LLVM in a little under 7 minutes, and, I didn't time it, but it can build the riscv gcc toolchain very quickly as well. My current (soon to be previous) dev box took about an hour to compile LLVM (if it didn't fail linking due to running out of memory) so doing any kind of LLVM development or making changes to binutils was quite tedious.

Post reply on HN