161–170 of 222 posts
The mainstream options seem to be Ryzen AI Max 395+, ~120 tops (fp8?), 128GB RAM, $1999 Nvidia DGX Spark, ~1000 tops fp4, 128GB RAM, $3999 Mac Studio max spec, ~120 tflops (fp16?), 512GB RAM, 3x bandwidth, $9499 DGX Spark appears to potentially offer the most token per second, but less useful/value as everyday pc.
suppose 1/3rd of memory is used to host a teacher network, and 2/3rds of memory is used to host a student network, how long would knowledge distillation typically take?
Earlier quoted context omitted.
Ok then just to clarify: you can fit 4x larger models on the Spark vs 5090, not 17x.
@nabla9 have tried to tell you that for DGX Spark, you can also use optimized models; therefore, this means that Spark can also be used for inference with bigger models, such as those exceeding 200B. Please compare the same things: carrots VS carrots, not apples VS eggs.
The RAM bandwidth is so slow on this that you can barely train or do inference or do anything on it. I think the only use case they have in mind for this is fine tuning pretrained models.
It's the same as Strix Halo and M4 Max that people are going gaga about, so either everyone is wrong or it's fine.
Hence the disappointment.
According to Wendell from Level1Techs, the now-launched Jetson Thor uses a Linux Kernel built by Nvidia, on Ubuntu 20.04 [0]. So I assume getting upgrades will have the same feel as those Chinese SBC's like from Radxa or cheap Android devices. I wonder if this also applies to this DGX Spark. I hope not. [0] https://www.youtube.com/watch?v=cgnKUUcCKcs&t=669s
That platform was great for a few months...
It's a great idea that they utterly gimped with the 273 GB/s memory bandwidth. You can buy a Macbook Pro M4 Max with 128GB that has TWICE the bandwidth. A 4090 has 4 TIMES the bandwidth. I get that they're terrified of doing anything that competes with their H100/B200 but there is an enormous gap between 273 GB/s and the >2TB/s of their data center products.