Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

NVIDIA DGX Spark In-Depth Review: A New Standard for Local AI Inference

lmsys.org

Post reply on HN
Print view View original

11–20 of 100 posts

Re: NVIDIA DGX Spark In-Depth Review: A New Standard for Local AI Inference

#11

Post by yvbbrjdr »

ggerganov wrote: post #10
yvbbrjdr wrote: post #8

Earlier quoted context omitted.

Hi, author here. I crowd-sourced the devices for benchmarking from my friends. It just happened that one of my friend has this device.

FYI you should have used llama.cpp to do the benchmarks. It performs almost 20x faster than ollama for the gpt-oss-120b model. Here are some samples results on my spark: ggml_cuda_init: found 1 CUDA devices: Device 0: NVIDIA GB10, compute capability 12.1, VMM: yes | model | size | params | backend | ngl | n_ubatch | fa | test | t/s | | ------------------------------ | ---------: | ---------: | ---------- | --: | ----…

I see! Do you know what's causing the slowdown for ollama? They should be using the same backend..

Re: NVIDIA DGX Spark In-Depth Review: A New Standard for Local AI Inference

#12

Post by moondev »

nialse wrote: post #4
SethTro wrote: post #2

Article doesn't seem to mention price which is $4,000 which makes it comparable to a 5090 but with 128GB of unified LPDDR5x vs the 5090's 32GB DDR7.

Well, that’s disappointing since the Mac Studio 128GB is $3,499. If Apple happens to launch a Mac Mini with 128GB RAM it would eat Nvidia Sparks’ lunch every day.

Just don't try to run a NCCL

Re: NVIDIA DGX Spark In-Depth Review: A New Standard for Local AI Inference

#13

Post by __mharrison__ »

ggerganov wrote: post #10
yvbbrjdr wrote: post #8

Earlier quoted context omitted.

Hi, author here. I crowd-sourced the devices for benchmarking from my friends. It just happened that one of my friend has this device.

FYI you should have used llama.cpp to do the benchmarks. It performs almost 20x faster than ollama for the gpt-oss-120b model. Here are some samples results on my spark: ggml_cuda_init: found 1 CUDA devices: Device 0: NVIDIA GB10, compute capability 12.1, VMM: yes | model | size | params | backend | ngl | n_ubatch | fa | test | t/s | | ------------------------------ | ---------: | ---------: | ---------- | --: | ----…

Curious to how this compares to running on a Mac.

Re: NVIDIA DGX Spark In-Depth Review: A New Standard for Local AI Inference

#15

Post by EnPissant »

pixelpoet wrote: post #5

I wonder why they didn't test against the broadly available Strix Halo with 128GB of 256 GB/s memory bandwidth, 16 core full-fat Zen5 with AVX512 at $2k... it is a mystery...

Strix Halo has the problem that prefill is incredibly slow if your context is not very small.

The only thing that might be interesting about this DGX Spark is it's prefill manages to be faster due to better compute. I haven't compared the numbers yet, but they are included in the article.

Re: NVIDIA DGX Spark In-Depth Review: A New Standard for Local AI Inference

#16

Post by rajatgupta314 »

ggerganov wrote: post #10
yvbbrjdr wrote: post #8

Earlier quoted context omitted.

Hi, author here. I crowd-sourced the devices for benchmarking from my friends. It just happened that one of my friend has this device.

FYI you should have used llama.cpp to do the benchmarks. It performs almost 20x faster than ollama for the gpt-oss-120b model. Here are some samples results on my spark: ggml_cuda_init: found 1 CUDA devices: Device 0: NVIDIA GB10, compute capability 12.1, VMM: yes | model | size | params | backend | ngl | n_ubatch | fa | test | t/s | | ------------------------------ | ---------: | ---------: | ---------- | --: | ----…

Is this the full weight model or quantized version? The GGUFs distributed on Hugging Face labeled as MXFP4 quantization have layers that are quantized to int8 (q8_0) instead of bf16 as suggested by OpenAI.

Example looking at blk.0.attn_k.weight, it's q8_0 amongst other layers:

https://huggingface.co/ggml-org/gpt-oss-20b-GGUF/tree/main?s...

Example looking at the same weight on Ollama is BF16:

https://ollama.com/library/gpt-oss:20b/blobs/e7b273f96360

Re: NVIDIA DGX Spark In-Depth Review: A New Standard for Local AI Inference

#17

Post by EnPissant »

yvbbrjdr wrote: post #9
pixelpoet wrote: post #5

I wonder why they didn't test against the broadly available Strix Halo with 128GB of 256 GB/s memory bandwidth, 16 core full-fat Zen5 with AVX512 at $2k... it is a mystery...

Hi, author here. I crowd-sourced the devices for benchmarking from my friends. It just happened that none of my friend has this device.

Something is wrong with your numbers: gpt-oss-20b and gpt-oss-120b should be much much faster than what you are seeing. I would suggest you familiarize yourself with llama-bench instead of ollama.

Running gpt-oss-120b with a rtx 5090 and 2/3 of the experts offloaded to system RAM (less than half of the memory bandwidth of this thing), my machine gets ~4100tps prefill and ~40tps decode.

Your spreadsheet shows the spark getting ~94tps prefill and ~11tps decode.

Now, it's expected that my machine should slaughter this thing in prefill, but decode should be very similar or the spark a touch faster.

Re: NVIDIA DGX Spark In-Depth Review: A New Standard for Local AI Inference

#18

Post by hank808 »

You guys that continue to compare DGX Spark to the Mac Studios, please remember two things:

1. Virtually every model that you'd run was developed on Nvidia gear and will run on Spark. 2. Spark has fast-as-hell interconnects. The sort of interconnects that one would want to use in an actual AI DC, so you can use more than one Spark at the same time, and RDMA, and actually start to figure out how things work the way they do and why. You can do a lot with 200 Gb of interconnect.

Re: NVIDIA DGX Spark In-Depth Review: A New Standard for Local AI Inference

#19

Post by [deleted] »

__mharrison__ wrote: post #13
ggerganov wrote: post #10

Earlier quoted context omitted.

FYI you should have used llama.cpp to do the benchmarks. It performs almost 20x faster than ollama for the gpt-oss-120b model. Here are some samples results on my spark: ggml_cuda_init: found 1 CUDA devices: Device 0: NVIDIA GB10, compute capability 12.1, VMM: yes | model | size | params | backend | ngl | n_ubatch | fa | test | t/s | | ------------------------------ | ---------: | ---------: | ---------- | --: | ----…

Curious to how this compares to running on a Mac.

[deleted]

Re: NVIDIA DGX Spark In-Depth Review: A New Standard for Local AI Inference

#20

Post by yvbbrjdr »

EnPissant wrote: post #17
yvbbrjdr wrote: post #9

Earlier quoted context omitted.

Hi, author here. I crowd-sourced the devices for benchmarking from my friends. It just happened that none of my friend has this device.

Something is wrong with your numbers: gpt-oss-20b and gpt-oss-120b should be much much faster than what you are seeing. I would suggest you familiarize yourself with llama-bench instead of ollama. Running gpt-oss-120b with a rtx 5090 and 2/3 of the experts offloaded to system RAM (less than half of the memory bandwidth of this thing), my machine gets ~4100tps prefill and ~40tps decode. Your spreadsheet shows the spar…

We actually profiled one of the models, and saw that the last GeMM, which is completely memory bound, is taking a lot of time, which reduces the token speed by a lot.
Post reply on HN