Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

NVIDIA DGX Spark In-Depth Review: A New Standard for Local AI Inference

lmsys.org

Post reply on HN
Print view View original

81–90 of 100 posts

Re: NVIDIA DGX Spark In-Depth Review: A New Standard for Local AI Inference

#81

Post by wmf »

miladyincontrol wrote: post #75
nialse wrote: post #65

Earlier quoted context omitted.

Also remember that the Mx Ultras have 2-3x the memory bandwidth. Looking at the benchmarks even Strix Halo seems to beat the Spark. Buying a 200 Gbps switch is $10k-$100k+ so don't imagine anyone actually will use the interconnect. The logical thing for Nvidia would be to sell a kit with three machines and cabling, and make it a ring with the dual ports per machine. Helps for some scenarios but not others with the 10…

On another note to remember, you can also ring topology mac studios using TB5 for 120Gbps per link with four such ports, all using cheaply available cable

You could also connect Sparks in a 200 Gbps ring with cheapish ($90) cables.

Re: NVIDIA DGX Spark In-Depth Review: A New Standard for Local AI Inference

#82

Post by hoppp »

arresin wrote: post #70
aurareturn wrote: post #41

It isn't that good for local LLM inferencing. It's not designed to be as such. It's designed to be a local dev machine for Nvidia server products. It has the same software and hardware stack as enterprise Nvidia hardware. That's what it is designed for. Wait for M5 series Macs for good value local inferencing. I think the M5 Pro/Max are going to be very good values.

I wish I could run Linux on them (the m5)

There was an arch linux version that supports apple silicon

Re: NVIDIA DGX Spark In-Depth Review: A New Standard for Local AI Inference

#83

Post by zackangelo »

moondev wrote: post #12
nialse wrote: post #4

Earlier quoted context omitted.

Well, that’s disappointing since the Mac Studio 128GB is $3,499. If Apple happens to launch a Mac Mini with 128GB RAM it would eat Nvidia Sparks’ lunch every day.

Just don't try to run a NCCL

Wouldn't you be able to test nccl if you had 2 of these?

Re: NVIDIA DGX Spark In-Depth Review: A New Standard for Local AI Inference

#84

Post by metadat »

Xss3 wrote: post #52
bilekas wrote: post #26

Earlier quoted context omitted.

$4,000 is actually extremely competitive. Even for an at-home enthusiast setup this price is not our of reach. I was expecting something far higher, that said, nVidia's MSRP is something of a pipe dream recently so we'll see when it's actually released and the availability. Curious also to see how they may scale together.

A warning to any home consumer throwing money at hardware for AI (fair enough if you have other use cases)... Things are changing rapidly and there is a non insignificant chance that it'll seem like a big waste of money within 12 months.

Based on what data? I'm not denying the possibility but this seems like baseless FUD. We haven't even seen what folks have done with this hardware yet.

Re: NVIDIA DGX Spark In-Depth Review: A New Standard for Local AI Inference

#85

Post by Eggpants »

xs83 wrote: post #40
__mharrison__ wrote: post #13

Earlier quoted context omitted.

Curious to how this compares to running on a Mac.

TTFT on a Mac is terrible and only increases as the context increases, thats why many are selling their M3 Ultra 512GB

So so many… eBay search shows only 15 results, 6 of them being ads for new systems…

https://www.ebay.com/sch/i.html?_nkw=mac+studio+m3+ultra+512...

Re: NVIDIA DGX Spark In-Depth Review: A New Standard for Local AI Inference

#86

Post by christkv »

NaomiLehman wrote: post #54
aurareturn wrote: post #41

It isn't that good for local LLM inferencing. It's not designed to be as such. It's designed to be a local dev machine for Nvidia server products. It has the same software and hardware stack as enterprise Nvidia hardware. That's what it is designed for. Wait for M5 series Macs for good value local inferencing. I think the M5 Pro/Max are going to be very good values.

because of possible hardware-accelerated matmul in GPU cores?

Massive memory bandwidth for the most part. M3 ultra had like 810 Gb/s vs ~300 for the DGX Spark. Also you can get up to 512 GB memory with a 256 GB config as well

Re: NVIDIA DGX Spark In-Depth Review: A New Standard for Local AI Inference

#87

Post by spaceywilly »

aurareturn wrote: post #41

It isn't that good for local LLM inferencing. It's not designed to be as such. It's designed to be a local dev machine for Nvidia server products. It has the same software and hardware stack as enterprise Nvidia hardware. That's what it is designed for. Wait for M5 series Macs for good value local inferencing. I think the M5 Pro/Max are going to be very good values.

What is the value proposition for buying one of these vs renting time on similar hardware from a cloud provider?

Re: NVIDIA DGX Spark In-Depth Review: A New Standard for Local AI Inference

#88

Post by moondev »

zackangelo wrote: post #83
moondev wrote: post #12

Earlier quoted context omitted.

Just don't try to run a NCCL

Wouldn't you be able to test nccl if you had 2 of these?

Not with Mac studio(s), but yes multi host NCCL over RoCE with two DGX Sparks or over PCI with one

Re: NVIDIA DGX Spark In-Depth Review: A New Standard for Local AI Inference

#89

Post by jamesblonde »

aurareturn wrote: post #41

It isn't that good for local LLM inferencing. It's not designed to be as such. It's designed to be a local dev machine for Nvidia server products. It has the same software and hardware stack as enterprise Nvidia hardware. That's what it is designed for. Wait for M5 series Macs for good value local inferencing. I think the M5 Pro/Max are going to be very good values.

Given that most of Nvidia's enterprise software products are all single server designed to run on DGX boxes, like NIMs, this makes sense.

I am still amazed at how many companies buy a ton of DGX boxes and then are surprised that Nvidia does not have any Kubernetes native platform for training and inferencing across all the DGX machines. The Run.ai acquisition did not change anything, as you leave all the work to the user to integrate with distributed training frameworks like Ray or scalable inference platforms, like KServe/vLLM.

Re: NVIDIA DGX Spark In-Depth Review: A New Standard for Local AI Inference

#90

Post by porphyra »

hoppp wrote: post #82
arresin wrote: post #70

Earlier quoted context omitted.

I wish I could run Linux on them (the m5)

There was an arch linux version that supports apple silicon

Asahi Linux has long switched from Arch to Fedora (though a janky Arch version still exists). But they don't support anything newer than M2.
Post reply on HN