Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Nvidia DGX Spark

nvidia.com

Post reply on HN
Print view View original

191–200 of 222 posts

Re: Nvidia DGX Spark

#191

Post by eadwu »

gardnr wrote: post #70
eadwu wrote: post #56

Most people are missing the point. LLMs are not the be all end all of AI. Even if you were to say memory bandwidth was the problem, there is no consumer grade GPU that can run any SoTA LLM, no matter what you'd have to settle for a more mediocre model. Outside of LLMs, 256 GB/s is not as much of an issue and many people have dealt with less bandwidth for real world use cases.

What other use cases would use 128GB VRAM but not require higher throughput to run at acceptable speeds?

People have done more with less for a long time (basically with the Jetson counterparts).

The only likely difference with DGX Spark is that it'll be a more desktop-centered platform, what people can do with it, not sure, but say for VR, the DGX Spark is basically the best compute puck for one right now.

Re: Nvidia DGX Spark

#192

Post by throwaway48476 »

aurareturn wrote: post #180
throwaway48476 wrote: post #173

Earlier quoted context omitted.

Its not priced in a linear way wrt to bom cost.

That's because memory channels cost money. Memory controllers are more complex. Lastly, chips that can make use of high bandwidth VRAM are both of the above. Memory chips are a commodity, that I agree. Though HBM is trending towards not being a commodity.

Memory controllers are die area. mm2 die space is linear bom cost.

DRAM+mm2 bom will have a different slope to just DRAM bom but still basically linear. Nonlinear pricing is pure market segmentation.

Re: Nvidia DGX Spark

#193

Post by dheera »

jacquesm wrote: post #150
aurareturn wrote: post #50

Earlier quoted context omitted.

It's not good value when you put it like that. It doesn't have a lot of compute and bandwidth. What it has is the ability to run DGX software for CUDA devs I guess. Not a great inference machine either.

It's great at one thing: memory. And that's interesting because memory is a commodity, but they still make bank on just being able to access it.

The RTX Pro 6000 Blackwell gets you 96GB of RAM, a LOT more compute, and costs ~$7K which is not that much more than the $3K-$4K you'd pay for a DGX Spark.

I think the RTX Pro is probably the best deal right now if you're looking for a GPU dev desktop and don't care about physical size or power consumption.

Re: Nvidia DGX Spark

#194

Post by magicalhippo »

linuxftw wrote: post #189
alias_neo wrote: post #143

Earlier quoted context omitted.

I'm pretty new to this, so if I wanted to benchmark my current hardware and compare to your results what would be the best way to do that? I'm looking at going for a Framework Desktop and would like to know what kind of performance gain I'd get over the current hardware I have, which so far I have a "feel" for the performance of from running Ollama and OpenWebUI, but no hard numbers.

What nobody seems to ever share is the context and TTFT (time to first token). You can get a very good TPS by using small prompts, even if the output tokens are very large. If you try to do any kind of agentic coding locally, where contexts are 7k+, local hardware completely falls over. qwen-code (cli) gives like 2k requests per day for free (and is fantastic), so unless you have a very specific use case, buying a sy…

> If you try to do any kind of agentic coding locally, where contexts are 7k+, local hardware completely falls over.

With my 5070 Ti + 2080 Ti I have Qwen 3 Coder 30B Q4_K_M running entirely on the GPUs with 16k context. Not great for larger code bases, but not nothing either.

Asking it to summarize llama-model-loader.cpp, which is about ~12k tokens, the TTFT is ~13 seconds and generation speed is about 55 tok/sec.

So yeah, for local stuff it's pick any two of large models, long contexts and decent speed.

Re: Nvidia DGX Spark

#195

Post by linuxftw »

magicalhippo wrote: post #194
linuxftw wrote: post #189

Earlier quoted context omitted.

What nobody seems to ever share is the context and TTFT (time to first token). You can get a very good TPS by using small prompts, even if the output tokens are very large. If you try to do any kind of agentic coding locally, where contexts are 7k+, local hardware completely falls over. qwen-code (cli) gives like 2k requests per day for free (and is fantastic), so unless you have a very specific use case, buying a sy…

> If you try to do any kind of agentic coding locally, where contexts are 7k+, local hardware completely falls over. With my 5070 Ti + 2080 Ti I have Qwen 3 Coder 30B Q4_K_M running entirely on the GPUs with 16k context. Not great for larger code bases, but not nothing either. Asking it to summarize llama-model-loader.cpp, which is about ~12k tokens, the TTFT is ~13 seconds and generation speed is about 55 tok/sec. S…

Yeah, that sounds decent for some one-shots. The unified memory systems can have longer back-and-forth context chats, but at slower speed (at least on AMD).

I find Qwen 3 Coder to be quite usable, I get around 20TPS on my AMD AI 350 system, as long as the net-new context isn't too big.

Re: Nvidia DGX Spark

#196

Post by KingOfCoders »

spwa4 wrote: post #137
KingOfCoders wrote: post #87

Earlier quoted context omitted.

In Germany the 96gb version is 5000 EUR and the 256gb version is 7000 EUR (no 128gb available as far as I can see).

At that point it's far superior to fly to the US, buy it, and fly back. Hell, have a nice week in a hotel and bring two.

Doesn't work with Germany, you'll have to pay VAT at the airport.

Re: Nvidia DGX Spark

#197

Post by Gracana »

apitman wrote: post #58
dirtyhand wrote: post #14

I was considering getting an RTX 5090 to run inference on some LLM models, but now I’m wondering if it’s worth paying an extra $2K for this option instead

If you want to run small models fast get the 5090. If you want to run large models slow get the Spark. If you want to run small models slow get a used MI50. If you want to run large models fast get a lot more money.

You might be able to do "large models slow" better than the spark with a 5090 and CPU offload, so long as you stick with MoE architectures. With the kv cache and shared parts of the model on GPU and all of the experts on CPU, it can work pretty well. I'm able to run ~400GB models at 10 tps with some A4000s and a bunch of RAM. That's on a Xeon W system with poor practical memory bandwidth (~190GB/s), you can do better with EPYC.

Re: Nvidia DGX Spark

#198

Post by pjs_ »

So if I buy 1000 of these I have an exascale supercomputer? I remember when exascale was disparaged as science fiction :)

Re: Nvidia DGX Spark

#199

Post by jacquesm »

dheera wrote: post #193
jacquesm wrote: post #150

Earlier quoted context omitted.

It's great at one thing: memory. And that's interesting because memory is a commodity, but they still make bank on just being able to access it.

The RTX Pro 6000 Blackwell gets you 96GB of RAM, a LOT more compute, and costs ~$7K which is not that much more than the $3K-$4K you'd pay for a DGX Spark. I think the RTX Pro is probably the best deal right now if you're looking for a GPU dev desktop and don't care about physical size or power consumption.

What I ended up doing is buy 3090's in bulk from ex mining rigs, stick new fans in them and run 14 of them connected to an old Supermicro chassis using very nice PCI express splitters. I figured out just what the sweet spot was in terms of PCI bandwidth and maxed it out on RAM (512G). The end result was pretty usable and allowed me to run some of my own benchmarks without sharing any data with the usual suspects. The bigger problem was power delivery, the box had four pretty beefy power supplies running on two different phases. It's retired now because we're much too busy with other stuff but that was a lot of fun and gave me a much better angle on all these new developments than I would have gotten otherwise. Total cost of the system was less than one RTX Pro 6000 and it had a lot more VRAM. The whole thing looked terrible though :)

Re: Nvidia DGX Spark

#200

Post by nodesocket »

KingOfCoders wrote: post #196
spwa4 wrote: post #137

Earlier quoted context omitted.

At that point it's far superior to fly to the US, buy it, and fly back. Hell, have a nice week in a hotel and bring two.

Doesn't work with Germany, you'll have to pay VAT at the airport.

What are they examining your bags for purchased items? That’s gestapoish.
Post reply on HN