Earlier quoted context omitted.
Just don't try to run a NCCL
Wouldn't you be able to test nccl if you had 2 of these?
What kind of NCCL testing are you thinking about? Always curious what’s hardest to validate in people’s setups.
NVIDIA DGX Spark In-Depth Review: A New Standard for Local AI Inference
91–100 of 100 posts
Re: NVIDIA DGX Spark In-Depth Review: A New Standard for Local AI Inference
#92It isn't that good for local LLM inferencing. It's not designed to be as such. It's designed to be a local dev machine for Nvidia server products. It has the same software and hardware stack as enterprise Nvidia hardware. That's what it is designed for. Wait for M5 series Macs for good value local inferencing. I think the M5 Pro/Max are going to be very good values.
because of possible hardware-accelerated matmul in GPU cores?
Yes. Matmul in M5 GPU, memory bandwidth, consumer/prosumer friendly OS, and they are just excellent portable laptops.
Re: NVIDIA DGX Spark In-Depth Review: A New Standard for Local AI Inference
#93Earlier quoted context omitted.
More precisely, the RTX 5090 has a memory bandwidth of 1792 GB/s, while the DGX Spark only has 273 GB/s, which is about 1/6.5. For inference, the DGX Spark does not look like a good choice, as there are cheaper alternatives with better performance.
My understanding is that the Jetson Thor is just as good a platform, and likely more readily available. Then there's the Mac Studio, which outdoes them in all respects except FP8 and FP4 support. As someone on Reddit put it: https://old.reddit.com/r/LocalLLaMA/comments/1n0xoji/why_can...
The jetson thor seems to be quite different. The Thor whitepaper lists 8 TFlop/s of FP32 compute where the DGX sparks seems to be closer to 30 TFlop/s. Also 48 SMs on the Spark vs 20 on the Jetson.
The DGX seems vastly more capable.
Re: NVIDIA DGX Spark In-Depth Review: A New Standard for Local AI Inference
#94Article doesn't seem to mention price which is $4,000 which makes it comparable to a 5090 but with 128GB of unified LPDDR5x vs the 5090's 32GB DDR7.
$4,000 is actually extremely competitive. Even for an at-home enthusiast setup this price is not our of reach. I was expecting something far higher, that said, nVidia's MSRP is something of a pipe dream recently so we'll see when it's actually released and the availability. Curious also to see how they may scale together.
If you compare DGX Spark with Ryzen AI Max 395, do you still think that $4000 for the NVidia device is very competitive?
To me it seems like you're paying more than twice the price mostly for CUDA compatibility.
Re: NVIDIA DGX Spark In-Depth Review: A New Standard for Local AI Inference
#95That memory bandwidth choked out their performance. How can you claim 1000 tflops if it's not capable of delivering it. Seems they chose to sandbag the spark in favour of the rtx pro 6000. I guess my next one I'm looking out for is the Orange Pi AI studio pro. Should have 192gb of ram, so able to run qwen3 235b, even though it's ddr4, it's nearly double the bandwidth of the spark.
Good luck with any kind of coherent ecosystem and support. Also, if you're in the U.S., there is a good chance you'll get hit with tariffs which would wipe out any potential value. I'd much rather stick with nVidia that has an ecosystem (even Apple for that matter), than touch a system like this off of Alibaba.
>Good luck with any kind of coherent ecosystem and support.
Admittedly I'm not a huge fan of debian; likely would end up going Arch on this one.
>Also, if you're in the U.S.,
Im not.
> I'd much rather stick with nVidia that has an ecosystem (even Apple for that matter), than touch a system like this off of Alibaba.
I get that. Realistically I'm waiting for medusa halo, some affordable datacenter card, something.
Re: NVIDIA DGX Spark In-Depth Review: A New Standard for Local AI Inference
#96Just get 5 Mac minis.
would it perform better?
Re: NVIDIA DGX Spark In-Depth Review: A New Standard for Local AI Inference
#97It isn't that good for local LLM inferencing. It's not designed to be as such. It's designed to be a local dev machine for Nvidia server products. It has the same software and hardware stack as enterprise Nvidia hardware. That's what it is designed for. Wait for M5 series Macs for good value local inferencing. I think the M5 Pro/Max are going to be very good values.
What is the value proposition for buying one of these vs renting time on similar hardware from a cloud provider?
I don't think there is one. Honestly this version 1 is dead on arrival.
Re: NVIDIA DGX Spark In-Depth Review: A New Standard for Local AI Inference
#98It isn't that good for local LLM inferencing. It's not designed to be as such. It's designed to be a local dev machine for Nvidia server products. It has the same software and hardware stack as enterprise Nvidia hardware. That's what it is designed for. Wait for M5 series Macs for good value local inferencing. I think the M5 Pro/Max are going to be very good values.
Fascinating that we didn't have to wait too long. Apple announced M5 this morning. Does it compare though?
Re: NVIDIA DGX Spark In-Depth Review: A New Standard for Local AI Inference
#99Bruh, if it were priced at like $2,499 it would make sense, but this is just too much.
Re: NVIDIA DGX Spark In-Depth Review: A New Standard for Local AI Inference
#100It isn't that good for local LLM inferencing. It's not designed to be as such. It's designed to be a local dev machine for Nvidia server products. It has the same software and hardware stack as enterprise Nvidia hardware. That's what it is designed for. Wait for M5 series Macs for good value local inferencing. I think the M5 Pro/Max are going to be very good values.
If I understand correctly the DGX is for the development, and the AGX Thor is more geared toward local LLM inferencing [1],[2].
[1] (Updated) NVIDIA Jetson AGX Thor Developer Kit to Launch in Mid-August with 2070 TFLOPS AI Performance, Priced at $3499:
https://linuxgizmos.com/updated-nvidia-jetson-agx-thor-devel...
[2] AAEON Announces BOXER-8741AI with NVIDIA Jetson Thor T5000 Module:
https://linuxgizmos.com/aaeon-announces-boxer-8741ai-with-nv...