201–210 of 222 posts
Earlier quoted context omitted.
> If you try to do any kind of agentic coding locally, where contexts are 7k+, local hardware completely falls over. With my 5070 Ti + 2080 Ti I have Qwen 3 Coder 30B Q4_K_M running entirely on the GPUs with 16k context. Not great for larger code bases, but not nothing either. Asking it to summarize llama-model-loader.cpp, which is about ~12k tokens, the TTFT is ~13 seconds and generation speed is about 55 tok/sec. S…
Yeah, that sounds decent for some one-shots. The unified memory systems can have longer back-and-forth context chats, but at slower speed (at least on AMD). I find Qwen 3 Coder to be quite usable, I get around 20TPS on my AMD AI 350 system, as long as the net-new context isn't too big.
Need a big case tho or go bitcoin miner style.
Not seriously thinking about it, just playing around.
"developers can prototype, fine-tune, and inference [AI models]"... shouldn't it be infer ?
No. It's quite common for technical slang to deviate from general vocabulary. Cf. "compute" is a verb for normal people, but for techies it is also "hardware resources used to compute things".
Earlier quoted context omitted.
To me, what I think they are saying is that the Spark can use a FP16 unoptimized model with 200B parameters. However I don't really know.
You can't. The Spark has 128GB VRAM; the highest you can go in FP16 is 64B — and that's with no space for context. 200B is probably a rough estimate of Q4 + some space for context. The Spark has 4x the VRAM of a 5090. That's all you need to know from a "how big can it go" perspective.
With 128 GB of unified system memory, developers can experiment, fine-tune, or inference models of up to 200B parameters. Plus, NVIDIA ConnectX™ networking can connect two NVIDIA DGX Spark supercomputers to enable inference on models up to 405B parameters.Earlier quoted context omitted.
384GB is 75% of 512GB. The M3 Ultra bandwidth is over 800GB/s, though potentially less in practice. Using an M3 Ultra I think the performance is pretty remarkable for inference and concerns about prompt processing being slow in particular are greatly exaggerated. Maybe the advantage of the DGX Spark will be for training or fine tuning.
I very consistently see people say prompt processing is slow for larger context sizes ("notoriously slow"), something that is much less of an issue with eg CUDA setups.
Earlier quoted context omitted.
You can't. The Spark has 128GB VRAM; the highest you can go in FP16 is 64B — and that's with no space for context. 200B is probably a rough estimate of Q4 + some space for context. The Spark has 4x the VRAM of a 5090. That's all you need to know from a "how big can it go" perspective.
from the NVidia DGX Spark datasheet: With 128 GB of unified system memory, developers can experiment, fine-tune, or inference models of up to 200B parameters. Plus, NVIDIA ConnectX™ networking can connect two NVIDIA DGX Spark supercomputers to enable inference on models up to 405B parameters.
You can do it, if you quantize to FP4 — and Nvidia's special variant of FP4, NVFP4, isn't too bad (and it's optimized on Blackwell). Some models are even trained at FP4 these days, like the gpt-oss models. But gigabytes are gigabytes, and you can't squeeze 400GB of FP16 weights into only 128GB (or 256GB) of space.
The datasheet is telling you the truth: you can fit a 200B model. But it's not saying you can do that at FP16 — because you can't. You can only do it at FP4.
Earlier quoted context omitted.
That's because memory channels cost money. Memory controllers are more complex. Lastly, chips that can make use of high bandwidth VRAM are both of the above. Memory chips are a commodity, that I agree. Though HBM is trending towards not being a commodity.
Memory controllers are die area. mm2 die space is linear bom cost. DRAM+mm2 bom will have a different slope to just DRAM bom but still basically linear. Nonlinear pricing is pure market segmentation.
Memory controllers are die area. mm2 die space is linear bom cost.
Earlier quoted context omitted.
Doesn't work with Germany, you'll have to pay VAT at the airport.
What are they examining your bags for purchased items? That’s gestapoish.
2. You should read up on the Gestapo
Earlier quoted context omitted.
Once the updated Mac Studio with M4/M5 Ultra comes out, pretty much going to make the DGX irrelevant right?
If that would be true why aren't Mac sales banned in China instead of Nvidia GPUs?
Earlier quoted context omitted.
Memory controllers are die area. mm2 die space is linear bom cost. DRAM+mm2 bom will have a different slope to just DRAM bom but still basically linear. Nonlinear pricing is pure market segmentation.
Memory controllers are die area. mm2 die space is linear bom cost. Ignoring the design and platform support that comes with higher bandwidth memory controllers.
I don't think you know how industry pricing works. Wafers have a price, double mm2, double the price of chip in bom.