It's going to be amazing. Almost twice as fast for only 10 times the heat. Consumers aren't concerned with efficiency they only care about performance.
Nvidia is proposing a beast of a CPU system for Windows PCs
401–410 of 581 posts
Re: Nvidia is proposing a beast of a CPU system for Windows PCs
#402Earlier quoted context omitted.
> The reality is even cutting edge games and consumer workloads don’t actually take full use of the PCIe bandwidth of the GPU or the bandwidth of its GDDR memory Game dev here. For anyone reading this - it’s not because we’re lazy, it’s because _it’s really hard to do_. One of the biggest differences between the current generation consoles and the current gen PCs is unified memory.
How much of that difficulty comes from the chosen game engine? I assume the engine is the primary factor in how resources are allocated.
I think much of the difficulty is just that, for example, the 1.8 TB/s of an RTX 5090 is a lot of bandwidth for a game to use. That's over 50,000 4k textures per second at 32bpp.
Re: Nvidia is proposing a beast of a CPU system for Windows PCs
#403I dont really get the hype with all the N1X thing when in reality this is the same almost 1 yr old GB10 that was released with the DGX Spark and proved to be quite a disappointment
Re: Nvidia is proposing a beast of a CPU system for Windows PCs
#404Earlier quoted context omitted.
There’s something else. Memory size. Even if a Mac isn’t the fastest in raw numbers it may be faster if it can load the whole model in its ram (went up to 512 GB before shortages) than a couple 32 GB cards could with the data having to be constantly loaded over PCI-E. Because unified memory means the Apple GPUs can access all 512 GB at full speed. My understanding is this is the advantage that’s pushing huge Mac Stud…
You're correct about some things but mostly wrong. Yes, a Mac with 128GB+ will let you load some pretty big models. However, you're still not going to be able to run them at usable speeds. Here are some M5 Max benchmarks on a Qwen 27B model w/ 290K context.... 12 tokens/sec output. https://www.reddit.com/r/oMLX/comments/1swztoh/m5_max_128gb_... And that's a 27B model. So yes, a M5 Max 128GB will let you load some pre…
12 tok/s can absolutely be "usable output" depending on what you're doing. I agree though that the 27B dense model often feels slow due to an overall weakness of memory throughput on that particular platform. Most real-world 120B models though will be MoE-based with only a small fraction of active parameters, and these run quite well. Also, dense models can benefit from batching, which is at least marginally viable with Qwen if you stick to shorter contexts and smaller batches.
Re: Nvidia is proposing a beast of a CPU system for Windows PCs
#405Earlier quoted context omitted.
Ah. Well, what kind of consumer hardware/software combo could I purchase to use this? outside of perhaps the... PS4?
Everything that doesn't have a discrete GPU has unified memory these days. If you're asking for something closer to the RTX Spark or Apple Silicon then look at AMD's Strix Halo systems.
> Everything that doesn't have a discrete GPU has unified memory these days.
Sorry, I meant before the M1 came out. And you and I both know that "unified memory" doesn't refer to allocating ram to the gpu for zero-swap sharing.
Re: Nvidia is proposing a beast of a CPU system for Windows PCs
#406I can't really see wide adoption of local LLMs unless prices really start to climb. It makes sense to use cheaper hosted smaller models like Sonnet or even Kimi but these won't run a Kimi-class model and that is really the floor for non-toy agentic tasks. Spending 5k to avoid a $20 subscription really only makes sense for niche security reasons.
Re: Nvidia is proposing a beast of a CPU system for Windows PCs
#407Earlier quoted context omitted.
Ah. Well, what kind of consumer hardware/software combo could I purchase to use this? outside of perhaps the... PS4?
Every AMD APU since introduction of HSA did it, which is how AMD ended up doing SoCs for PS4, PS5, and Xbox
Ok, so which one of these contemporary or previous chipsets could compete with the M1 for inference? Perhaps I'm missing some major detail.
Re: Nvidia is proposing a beast of a CPU system for Windows PCs
#408I ran two gens of Jetson board and I have zero confidence in this. NVDA is printing in the data centre and everything else has no staying power.
Re: Nvidia is proposing a beast of a CPU system for Windows PCs
#409Earlier quoted context omitted.
Unified memory is only a feature because NVidia so aggressively uses VRAM for market segmentation. The 5090 ($2k MSRP but realistically $3-3.5k) is almost the same as the RTX 6000 Pro (~$10k). Same memory bandwidth (1800GB/s). Slightly different CUDA cores (21k vs 24k). Big difference? VRAM (32GB vs 96GB). NVidia ultimately doesn't want to upset this segmentation so the RTX Spark will never undermine their other offe…
What should Apple do, in your view, to "embrace" it?
Mx Extreme = 2 x Mx Ultra = more cores. (Opportunity: processor chiplets could be designed to integrate in higher quantities.)
Increase RDMA cross-bar linking from 4x to 8x = a lotta ports, a switch, or a stacking interface.
Regular RAM size/speed scaling: 512GB -> 1TB Mac Studios. Wider RAM and RDMA paths * clocks.
Given the low power envelope of today's Mac Studios, and bandwidth limits, lots of room to scale up, if Apple chooses. My fantasy: 2x cores, 2x RAM sizes, 2x RDMA devices, 2-4x RAM & RMDA bandwidth.
Re: Nvidia is proposing a beast of a CPU system for Windows PCs
#410Earlier quoted context omitted.
I'm not the person you're replying to, but I wholeheartedly agree with them... Quick background: doing AI inference requires three things. Lots of memory, lots of memory bandwidth, and of course plenty of compute that has access to that memory. Quick reference: nVidia 5090 has 1,792 GB/sec bandwidth. 3090 gets about 1000 GB/sec. DGX Spark and AMD 395 whatever get about 275 GB/sec. Apple M1 Max gets 400GB/sec, M5 Max…
They use different technology for their VRAM though. Apple, AMD Strix and NVidia DGX/RTX Spark use LPDDR, whereas discrete cards will be either GDDR or HBM. That directly impacts the memory bandwidth figures. As for compute available, Apple and AMD still have very good figures there for what's essentially a general-purpose iGPU that ships as part of the stock system, rather than a special-purpose piece of dedicated h…
The M5 has 16 dedicated ‘Neural Engine’ cores and a ‘Neural accelerator’ in each of its conventional GPU cores. It’s been pretty special-purpose juiced for inference.