131–140 of 149 posts
Nobody mentioning how this project is vibecoded slop? > The code is really bad with completely uneeded parts. The LLM (Qwen 2.5 7B) has hardcoded the i9 14700KF topology, and has variables related to it never used... It's even funnier that the show hardware function always prints the same string. There are even random pip log files. Why did this slop got coverage here? https://www.phoronix.com/forums/forum/linux-grap…
Earlier quoted context omitted.
You mean 500 GB/s, not Gb/s (actually 537 GB/s). Unfortunately that does not matter. Even in a cheap desktop motherboard the memory bandwidth is higher than of 16-lane PCIe 5.0. Therefore the memory bandwidth available to a discrete GPU is determined by its PCIe slot, not by the system memory. If you install multiple GPUs, in many MBs that will halve the bandwidth of the PCIe slots, for an even lower memory throughpu…
> in many MBs that will halve the bandwidth of the PCIe slots Not on boards that have 12 channels of DDR5. But yeah, squeezing an LLM from RAM through the PCIe bus is silly. I would expect it would be faster to just run a portion of the model on the CPU in llama.cop fashion.
Edit: the settings is "GGML_CUDA_ENABLE_UNIFIED_MEMORY=1"... useful if you have unified memory, very slow if you do not.
Why don't they just put ram slots on the card so you can augment the fast ram
Speed and reliability. A connector of any kind reduces signal quality. Data lines need to be longer, because the memory slot won't fit under the radiator where the memory chips are now, and that adds even more electrical interference and degrades signal. Also, we had memory slots on '90s cards. They were extremely expensive and proprietary. Ever saw a Matrox VRAM card? I never did.
Earlier quoted context omitted.
12 channel ddr5 5600 ECC is around 500gbs which in real world works very well for large MoE
You mean 500 GB/s, not Gb/s (actually 537 GB/s). Unfortunately that does not matter. Even in a cheap desktop motherboard the memory bandwidth is higher than of 16-lane PCIe 5.0. Therefore the memory bandwidth available to a discrete GPU is determined by its PCIe slot, not by the system memory. If you install multiple GPUs, in many MBs that will halve the bandwidth of the PCIe slots, for an even lower memory throughpu…
It’s fast for hybrid inference, if you get the KV and MoE layers tuned between the Blackwell card(s) and offloading.
We have a prototype unit and it’s very fast with large MoEs
Why don't they just put ram slots on the card so you can augment the fast ram
Speed and reliability. A connector of any kind reduces signal quality. Data lines need to be longer, because the memory slot won't fit under the radiator where the memory chips are now, and that adds even more electrical interference and degrades signal. Also, we had memory slots on '90s cards. They were extremely expensive and proprietary. Ever saw a Matrox VRAM card? I never did.
Earlier quoted context omitted.
On some workloads, swapping is a bad idea. The fundamental problem here is that the workload of LLMs is (vastly simplified) a repeated linear read of all the weights, in order. That is, there is no memory locality in time. There is literally anti-locality; When you read a set of weights, you know you will not need them again until you have processed everything else. This means that many of the old approaches don't wo…
The main blocker with swapping is not even the limited bandwidth, it's actually the extreme write workload on data elements such as the per-layer model activations - and, to a much lesser extent, the KV-cache. In contrast, there are elements such as inactive experts for highly sparse MoE models, where swapping makes sense since any given expert will probably be unused. You're better off using that VRAM/RAM for someth…
The activated experts is only available after routing, at which point you need the weights immediately and will have very poor performance if they are across PCIe
Why is there a new kernel driver here at all? It appears that all it does it allocate system RAM (“DDR4”) and export it as a dmabuf for import to cuda as mapped external memory. Then a userspace shim hijacks APIs to use that if gpu memory is full. cuda already supports allocating mapped system memory, so AFAICT this could be implemented in the userspace shim with no new kernel driver. Also as other commenters have me…
Or, as you said, making everything backwards compatible that is not being regularly updated
Earlier quoted context omitted.
The main blocker with swapping is not even the limited bandwidth, it's actually the extreme write workload on data elements such as the per-layer model activations - and, to a much lesser extent, the KV-cache. In contrast, there are elements such as inactive experts for highly sparse MoE models, where swapping makes sense since any given expert will probably be unused. You're better off using that VRAM/RAM for someth…
How do get the weights for the right set of experts for a given batch of tokens into fast memory at the right time? The activated experts is only available after routing, at which point you need the weights immediately and will have very poor performance if they are across PCIe
Earlier quoted context omitted.
Given that 32 GB/s is significantly worse than CPU to RAM speeds these days, does the additional compute really make it any faster in practice? The KV cache is always on the GPU anyway unless you're doing something really weird, so it won't affect ingestion, and generation is typically bandwidth bound. With something like ×16 PCIe 6.0 it would actually make sense, but nothing less than that, or maybe for smaller dens…
Additional compute is generally a win for prefill, while memory bandwidth is king for decode. KV cache however is the main blocker for long context, so it should be offloaded to system RAM and even to NVMe swap as context grows. Yes that's slow on an absolute basis but it's faster (and more power efficient, which makes everything else faster) than not having the cache at all, so it's still a huge win.
Earlier quoted context omitted.
Strix Halo uses a 256bit memory interface, the normal desktop processors only have a 128bit interface, that's the biggest difference in bandwidth. For more bandwidth you need to go to a Threadripper. Strix Halo seems to use LPDDR with 8000 MT/s, which is a bit faster than the usual 5600 MT/s-6400 MT/s "normal" DDR5-DIMMs (Albeit (expensive) faster ones seem to exist), so there's a slight edge towards soldered memory…
It's really hard to push DDR5 past 6000MT/s on 4+ DIMMs it seems. I had to get everything top spec to fit 4 channels of 6000MT/s on my 9950x (asus proArt motherboard and the top tier trident neo RAM sticks) -- otherwise it's reportedly unstable.
Strix Halo simply has more memory controllers. Threadrippers are also quad channel, and should be able to run 4 DIMMs at rated speeds, but the cheapest Zen 5 Threadripper seems to be almost double the price of a 9950X3D.
