Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Nvidia DGX Spark

nvidia.com

Post reply on HN
Print view View original

151–160 of 222 posts

Re: Nvidia DGX Spark

#151

Post by imtringued »

woooooo wrote: post #101
Rohansi wrote: post #64

Earlier quoted context omitted.

I would expect/hope that DGX would be able to make better use of its bandwidth than the M4 Max. Will need to wait and see benchmarks.

Matrix vector multiplication for feed forward layers is most of the bandwidth as I understand things, there's not really a way to do it "better", its just a bunch of memory-bound dot products. (Posting this comment in hopes of being corrected and learning something).

Training is performed in parallel with batching and is more flops heavy. I don't have an intuition on how memory bandwidth intensive updating the parameters is. It shouldn't be much worse than doing a single forward pass though.

Re: Nvidia DGX Spark

#152

Post by vid »

rz2k wrote: post #59
jauntywundrkind wrote: post #36

Earlier quoted context omitted.

The GPU memory allocation refers to how capacity is alloted, not bandwidth. Sounds like the same 256-bit/quad-channel 8000MHz lpddr5 you can get today with Strix Halo.

384GB is 75% of 512GB. The M3 Ultra bandwidth is over 800GB/s, though potentially less in practice. Using an M3 Ultra I think the performance is pretty remarkable for inference and concerns about prompt processing being slow in particular are greatly exaggerated. Maybe the advantage of the DGX Spark will be for training or fine tuning.

I very consistently see people say prompt processing is slow for larger context sizes ("notoriously slow"), something that is much less of an issue with eg CUDA setups.

Re: Nvidia DGX Spark

#153

Post by touisteur »

QQ00 wrote: post #126
skhameneh wrote: post #85

Earlier quoted context omitted.

It's more throughput, but way less value and there's still no NVLink on the 6000. Something like ~4x the price, ~20% more performance, 3x the VRAM. There's two models that go by 6000, the RTX Pro 6000 (Blackwell) is the one that's currently relevant.

the RTX Pro 6000 (Blackwell) does not have NVlink? if so, what the fuck Mr.leather jacket.

L40/L40S didn't have it either, which was announced late and felt like a gut-punch for the rare actual non-AI use-case at the time.

A40 nvlink was limited though, one-to-one (bridge), never saw it daisy-chained or nvswitched (might have just not seen them myself, they may have existed).

Re: Nvidia DGX Spark

#154

Post by syntaxing »

larodi wrote: post #89
syntaxing wrote: post #22

While a completely different price point, I have a Jetson Orin Nano. Some people forget the kernels are more or less set in stone for product like these. I could rebuild my own Jetpack kernel but it’s not that straight forward to update something like CUDA or any other module. Unless you’re a business where your product relies on this hardware, I find it hard to buy this for consumer applications.

My experience with Jetson Nano was that it had to have its Ubuntu debloatred first (with 3rd party script) before we could get their NN something library to run the image recognition, designated to run on this device. These seem to be highly experimental boards, even though are super powerful for their form factor.

That’s true for the Jetson Nano. The Jetson Orin Nano (I know, the naming sucks) is much better in that aspect. Higher memory (8GB), way higher memory bandwidth (120 GB/s), and Orin has way more CUDA cores. It can pretty much run any “traditional” neural network, even YOLO large and even LLMs.

Re: Nvidia DGX Spark

#155

Post by rowanG077 »

qwertox wrote: post #145

According to Wendell from Level1Techs, the now-launched Jetson Thor uses a Linux Kernel built by Nvidia, on Ubuntu 20.04 [0]. So I assume getting upgrades will have the same feel as those Chinese SBC's like from Radxa or cheap Android devices. I wonder if this also applies to this DGX Spark. I hope not. [0] https://www.youtube.com/watch?v=cgnKUUcCKcs&t=669s

Oooof that would be an instant dealbreaker for me... Better get a mac pro with asahi linux. That at least has great linux support.

Re: Nvidia DGX Spark

#156

Post by aurareturn »

themgt wrote: post #140
aurareturn wrote: post #132

Earlier quoted context omitted.

You'd have to stack 16 of these to get 2TB of VRAM, equivalent to 4 Mac Studios 512GBs chained together. 16 compared to 4. Surely even much faster networking in the Spark would degrade with that many devices? Biggest problem with Macs is that they don't have dedicated tensor cores in the GPU which makes prompt processing very slow compared to Nvidia and AMD.

n.b. there's been a little speculation that Apple adding TensorOps to Metal 4 suggests M5/M6 may get tensor cores. https://x.com/liuliu/status/1932158994698932505 https://developer.apple.com/metal/Metal-Shading-Language-Spe...

Nice. I hope so. That would make Macs the best local LLM machines for the masses by far.

Re: Nvidia DGX Spark

#157

Post by fh973 »

rowanG077 wrote: post #155
qwertox wrote: post #145

According to Wendell from Level1Techs, the now-launched Jetson Thor uses a Linux Kernel built by Nvidia, on Ubuntu 20.04 [0]. So I assume getting upgrades will have the same feel as those Chinese SBC's like from Radxa or cheap Android devices. I wonder if this also applies to this DGX Spark. I hope not. [0] https://www.youtube.com/watch?v=cgnKUUcCKcs&t=669s

Oooof that would be an instant dealbreaker for me... Better get a mac pro with asahi linux. That at least has great linux support.

Marketing material says NVIDIA DGX™ OS, which at version 7 would be an Ubuntu 24.04: https://docs.nvidia.com/dgx/dgx-os-7-user-guide/introduction...

Re: Nvidia DGX Spark

#158

Post by ThatMedicIsASpy »

rjzzleep wrote: post #92
cherioo wrote: post #24

The mainstream options seem to be Ryzen AI Max 395+, ~120 tops (fp8?), 128GB RAM, $1999 Nvidia DGX Spark, ~1000 tops fp4, 128GB RAM, $3999 Mac Studio max spec, ~120 tflops (fp16?), 512GB RAM, 3x bandwidth, $9499 DGX Spark appears to potentially offer the most token per second, but less useful/value as everyday pc.

Maybe the real value of the DGX spark is to work on Switch 2 emulation. ARM + Nvidia GPU. Start with Switch 2 emulation on this machine and then optimize for others. (Yeah, I know, kind of expensive toy).

Expensive today. But how quickly (years) will these systems lower in value? At least on the Nvidia side of things they can be stacked.. so maybe not so much =/

Re: Nvidia DGX Spark

#159

Post by aurareturn »

jacquesm wrote: post #150
aurareturn wrote: post #50

Earlier quoted context omitted.

It's not good value when you put it like that. It doesn't have a lot of compute and bandwidth. What it has is the ability to run DGX software for CUDA devs I guess. Not a great inference machine either.

It's great at one thing: memory. And that's interesting because memory is a commodity, but they still make bank on just being able to access it.

Memory is a commodity but access to high bandwidth memory is expensive whether it's HBM, or LPDDR/DDR connected to many memory channels.

Re: Nvidia DGX Spark

#160

Post by ilirium »

artemisart wrote: post #149
nabla9 wrote: post #148

Earlier quoted context omitted.

Its for comparison using raw, non optimized models. Both can do much better when you optimize for inference. Information is in the ratio of these numbers. They stay the same.

Ok then just to clarify: you can fit 4x larger models on the Spark vs 5090, not 17x.

@nabla9 have tried to tell you that for DGX Spark, you can also use optimized models; therefore, this means that Spark can also be used for inference with bigger models, such as those exceeding 200B.

Please compare the same things: carrots VS carrots, not apples VS eggs.

Post reply on HN