Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Nvidia DGX Spark

nvidia.com

Post reply on HN
Print view View original

101–110 of 222 posts

Re: Nvidia DGX Spark

#101

Post by woooooo »

Rohansi wrote: post #64
aurareturn wrote: post #51

Earlier quoted context omitted.

M4 max has more than double the bandwidth. Strix Halo has the same and I agree it’s overrated.

I would expect/hope that DGX would be able to make better use of its bandwidth than the M4 Max. Will need to wait and see benchmarks.

Matrix vector multiplication for feed forward layers is most of the bandwidth as I understand things, there's not really a way to do it "better", its just a bunch of memory-bound dot products.

(Posting this comment in hopes of being corrected and learning something).

Re: Nvidia DGX Spark

#102

Post by timc3 »

KingOfCoders wrote: post #88

I think it depends on your model size Fits into 32gb: 5090 Fits into 64gb - 96gb: Mac Studio Fits into 128gb: for now 395+ $/token/s, Mac Studio if you don't care about $ but don't have unlimited money for Hxxx This could be great for models that fit 128gb and you want best $/token/s (if it is faster than a 395+).

The 395 although it can be supplied with 128GB can’t use all that for VRAM (unless something has changed in the last couple of weeks).

Re: Nvidia DGX Spark

#103

Post by FirmwareBurner »

nodesocket wrote: post #68
hereme888 wrote: post #49

FP4-sparse (TFLOPS) | Price | $/TF4s 5090: 3352 | 1999 | 0.60 Thor: 2070 | 3499 | 1.69 Spark: 1000 | 3999 | 4.00 ____________ FP8-dense (TFLOPS) | Price | $/TF8d (4090s have no FP4) 4090 : 661 | 1599 | 2.42 4090 Laptop: 343 | vary | - ____________ Geekbench 6 (compute score) | Price | $/100k 4090: 317800 | 1599 | 503 5090: 387800 | 1999 | 516 M4 Max: 180700 | 1999 | 1106 M3 Ultra: 259700 | 3999 | 1540 ____________ Ap…

Once the updated Mac Studio with M4/M5 Ultra comes out, pretty much going to make the DGX irrelevant right?

If that would be true why aren't Mac sales banned in China instead of Nvidia GPUs?

Re: Nvidia DGX Spark

#104

Post by Auracle »

gardnr wrote: post #70
eadwu wrote: post #56

Most people are missing the point. LLMs are not the be all end all of AI. Even if you were to say memory bandwidth was the problem, there is no consumer grade GPU that can run any SoTA LLM, no matter what you'd have to settle for a more mediocre model. Outside of LLMs, 256 GB/s is not as much of an issue and many people have dealt with less bandwidth for real world use cases.

What other use cases would use 128GB VRAM but not require higher throughput to run at acceptable speeds?

Fine tuning text to image/video models perhaps?

For the newest models unless you quantize the crap out of them, even with a 5090 you’re going to be swapping blocks, which slows things down anyways. At least you’d be able to train on them at full precision with a decent batch size.

That said, I can’t imagine there’s enough of a market there to make it worth it.

Re: Nvidia DGX Spark

#105

Post by thomasskis »

nodesocket wrote: post #68
hereme888 wrote: post #49

FP4-sparse (TFLOPS) | Price | $/TF4s 5090: 3352 | 1999 | 0.60 Thor: 2070 | 3499 | 1.69 Spark: 1000 | 3999 | 4.00 ____________ FP8-dense (TFLOPS) | Price | $/TF8d (4090s have no FP4) 4090 : 661 | 1599 | 2.42 4090 Laptop: 343 | vary | - ____________ Geekbench 6 (compute score) | Price | $/100k 4090: 317800 | 1599 | 503 5090: 387800 | 1999 | 516 M4 Max: 180700 | 1999 | 1106 M3 Ultra: 259700 | 3999 | 1540 ____________ Ap…

Once the updated Mac Studio with M4/M5 Ultra comes out, pretty much going to make the DGX irrelevant right?

I run 4 Mac Studio ultras at work (they’re pricy when maxed out), for local-first AI dev services. But there’s a few things that make me want to switch to the Spark. Networking is the biggest one, the Macs have Thunderbolt and Ethernet, but if I run distributed inference with EXO over Thunderbolt; the drop in tokens/second is massive. These Sparks get RDMA and can stack nicely. The other big one is access to CUDA, MLX has come a long way but being able to have CUDA and GPU access in containers would simplify the stack so nicely. If I had a USB-C/Thunderbolt backplane it might compare, but scaling with the Spark is likely a lot more straightforward.

I call the stack with Mac Studios “MacAIver” because it feels like a duct tape solution, but the Spark equivalent would likely be more elegant.

Re: Nvidia DGX Spark

#106

Post by pta2002 »

rjzzleep wrote: post #92
cherioo wrote: post #24

The mainstream options seem to be Ryzen AI Max 395+, ~120 tops (fp8?), 128GB RAM, $1999 Nvidia DGX Spark, ~1000 tops fp4, 128GB RAM, $3999 Mac Studio max spec, ~120 tflops (fp16?), 512GB RAM, 3x bandwidth, $9499 DGX Spark appears to potentially offer the most token per second, but less useful/value as everyday pc.

Maybe the real value of the DGX spark is to work on Switch 2 emulation. ARM + Nvidia GPU. Start with Switch 2 emulation on this machine and then optimize for others. (Yeah, I know, kind of expensive toy).

I think you can get something a lot cheaper if that’s all you want, e.g. something in the Jetson Orin line. That’s more similar to the switch, also, since it’s a Tegra CPU.

Re: Nvidia DGX Spark

#107

Post by _zoltan_ »

wtallis wrote: post #96
fh973 wrote: post #93

Earlier quoted context omitted.

Ordered one in spring. Delivery time was pushed from July to September. Apparently they had a bug in the HDMI output.

That's eerily similar to what happened to Qualcomm's failed Snapdragon X Elite dev kit. That one eventually shipped in small quantities with a Type-C to HDMI dongle in the box to make up for the built-in HDMI port going missing. Then Qualcomm cancelled the whole project and refunded everyone, including people who had already received their hardware.

because they realized it sucked.

Re: Nvidia DGX Spark

#108

Post by nevi-me »

FirmwareBurner wrote: post #103
nodesocket wrote: post #68

Earlier quoted context omitted.

Once the updated Mac Studio with M4/M5 Ultra comes out, pretty much going to make the DGX irrelevant right?

If that would be true why aren't Mac sales banned in China instead of Nvidia GPUs?

Because Tim bribed Trump with a golden calf, or more seriously it's easier to ban a component and its manufacturer vs broader systems.

Re: Nvidia DGX Spark

#109

Post by lhl »

cherioo wrote: post #24

The mainstream options seem to be Ryzen AI Max 395+, ~120 tops (fp8?), 128GB RAM, $1999 Nvidia DGX Spark, ~1000 tops fp4, 128GB RAM, $3999 Mac Studio max spec, ~120 tflops (fp16?), 512GB RAM, 3x bandwidth, $9499 DGX Spark appears to potentially offer the most token per second, but less useful/value as everyday pc.

RDNA3 CUs do not have FP8 support and its INT8 runs at the same speed as FP16 so Strix Halo's max theoretical is basically 60 TFLOPS no matter how you slice it (well it has double INT4, but I'm unclear on how generally useful that is):

    512 ops/clock/CU * 40 CU * 2.9e9 clock / 1e12 = 59.392 FP16 TFLOPS
Note, even with all my latest manual compilation whistles and the latest TheRock ROCm builds the best I've gotten mamf-finder up to about 35 TFLOPS, which is still not amazing efficiency (most Nvidia cards are at 70-80%), although a huge improvement over the single-digit TFLOPS you might get ootb.

If you're not training, your inference speed will largely be limited by available memory bandwidth, so the Spark token generation will be about the same as the 395.

On general utility, I will say that the 16 Zen5 cores are impressive. It beats my 24C EPYC 9274F in single and multithreaded workloads by about 25%.

Post reply on HN