Hacked Nvidia 4090 GPU driver to enable P2P • Hacker News Board

Re: Hacked Nvidia 4090 GPU driver to enable P2P

#341

Post by Tepix » Sun, Apr 14, 2024, 12:13 AM UTC

andersa wrote: post #115

Tepix wrote: post #96

Earlier quoted context omitted.

6 GPUs because they want fast storage and it uses PCIe lanes. Besides the goal was to run a 70b FP16 model (requiring roughly 140GB VRAM). 6*24GB = 144GB

That calculation is incorrect. You need to fit both the model (140GB) and the KV cache (5GB at 32k tokens FP8 with flash attention 2) * batch size into VRAM. If the goal is to run a FP16 70B model as fast as possible, you would want 8 GPUs with P2P, for a total of 192GB VRAM. The model is then split across all 8 GPUs with 8-way tensor parallelism, letting you make use of the full 8TB/s memory bandwidth on every itera…

I know there's some overhead, it's not my calculation.

https://www.tweaktown.com/news/97110/tinycorps-new-tinybox-a...

Quote: "Runs 70B FP16 LLaMA-2 out of the box using tinygrad"

Re: Hacked Nvidia 4090 GPU driver to enable P2P

#342

Post by faeriechangling » Sun, Apr 14, 2024, 3:02 AM UTC

papichulo2023 wrote: post #337

faeriechangling wrote: post #335

Earlier quoted context omitted.

Most consumer mobo's I see support this even if the setup isn't on the QVL. If a DDR5 motherboard support 4 sticks at all you can probably run 192gb on it so long as you update the BIOS firmware. The problem is running at rated speeds. AMD tends to be worse than Intel, and I hear people having to run anywhere between DDR5-3200 to DDR5-5200. You are better off running two sticks, because even with 2 sticks you really…

Hopefully Qualcom will free us of this 2 channels noghtmare.

I’m optimistic about APUs personally like AMDs upcoming Strix Halo APU with a 256-bit memory bus competing at the lower end of the market, but that will only provide so much competition.

Re: Hacked Nvidia 4090 GPU driver to enable P2P

#343

Post by fulafel » Sun, Apr 14, 2024, 5:25 AM UTC

ot1138 wrote: post #35

__alexs wrote: post #34

Earlier quoted context omitted.

It means you can send data from the memory of 1 GPU to another GPU without going via RAM. https://xilinx.github.io/XRT/master/html/p2p.html

Is this really efficient or practical? My understanding is that the latency required to copy memory from CPU or RAM to GPU negates any performance benefits (much less running over a network!)

Yes, networking is similarly pointless.

Re: Hacked Nvidia 4090 GPU driver to enable P2P

#344

Post by numpad0 » Sun, Apr 14, 2024, 3:17 PM UTC

LoganDark wrote: post #256

int_19h wrote: post #249

Earlier quoted context omitted.

It doesn't beat RTX 4090 when it comes to actual LLM inference speed. I bought a Mac Studio for local inference because it was the most convenient way to get something fast enough and with enough RAM to run even 155b models. It's great for that, but ultimately it's not magic - NVidia hardware still offers more FLOPS and faster RAM.

> It doesn't beat RTX 4090 when it comes to actual LLM inference speed Sure, whisper.cpp is not an LLM. The 4090 can't even do inference at all on anything over 24GB, while ASi can chug through it even if slightly slower. I wonder if with https://github.com/tinygrad/open-gpu-kernel-modules (the 4090 P2P patches) it might become a lot faster to split a too-large model across multiple 4090s and still outperform ASi (at…

PSA for all people who are still being misled by hand-wavy Apple M1 marketing charts[1] implicating total dominance of M-series wondersilicon obsoleting all Intel/NVIDIA PCs:

There are benchmark data showing that an Apple M2 Ultra is 47% and 60% slower against Xeon W9 and RTX 4090, or 0.35% and 2% slower against i9-13900K and RTX 4060 Ti, respectively, in Geekbench 5 Multi-threaded and OpenCL Compute tests.

Apple Silicon Macs are NOT faster than competing desktop computers, nor M1 was massively faster than NVIDIA 3070(Desktop - 2x faster than Laptop variant M1 was compared against) for that matter. They just offer up to 128GB shared RAM/VRAM options in slim desktops and laptops, which is handy for LLM, that's it.

Please stop taking Apple marketing materials at full face value or above. Thank you.

  1: https://i.extremetech.com/imagery/content-types/03ekiQwNudC75iOK4AMuEkw/images-2.jpg    
  2: screenshot from[4]: https://www.igorslab.de/wp-content/uploads/2023/06/Apple-M2-ULtra-SoC-Geekbench-5-Multi-Threaded.jpg  
  3: screenshot from[4]: https://www.igorslab.de/wp-content/uploads/2023/06/Apple-M2-ULtra-SoC-Geekbench-5-OpenCL-Compute.jpg  
  4: https://wccftech.com/apple-m2-ultra-soc-isnt-faster-than-amd-intel-last-year-desktop-cpus-50-slower-than-nvidia-rtx-4080/

Re: Hacked Nvidia 4090 GPU driver to enable P2P

#345

Post by uniqueuid » Sun, Apr 14, 2024, 3:56 PM UTC

chriskanan wrote: post #316

This is great news. As an academic, I'm aware of multiple labs that built boxes with 4090s, not realizing that Nvidia had impaired P2P communication among cards. It's one of the reasons I didn't buy 4090s, despite them being much more affordable for my work. It isn't nvlink, but Nvidia has mostly gotten rid of that except for their highest end cards. It is better than nothing. Late last year, I got quotes for machine…

And before that, Nvidia made our lives harder by phasing out blower-style designs in consumer cards that we could put in servers. In my lab, I'd take a card for 1/4 the price that has half the MTBF over a card for full price anytime.

Re: Hacked Nvidia 4090 GPU driver to enable P2P

#346

Post by gururise » Sun, Apr 14, 2024, 4:10 PM UTC

How long before Nvidia patches this?

Re: Hacked Nvidia 4090 GPU driver to enable P2P

#347

Post by matheusmoreira » Sun, Apr 14, 2024, 4:37 PM UTC

userbinator wrote: post #76

I wish more hardware companies would publish more documentation and let the community figure out the rest, sort of like what happened to the original IBM VGA (look up "Mode X" and the other non-BIOS modes the hardware is actually capable of - even 800x600x16!) Sadly it seems the majority of them would rather tightly control every aspect of their products' usage since they can then milk the userbase for more $$$, but…

> the most productive era of the PC was also when it was the most open

The openness certainly was great but it's not actually required. People can figure out how to work with closed systems. Adversarial interoperability was common. People would reverse engineer things and make the software work whether or not the manufacturer wanted it.

It's the software and hardware locks that used to be rare and are now common. Cryptography was supposed to be something that would empower us but it ended up being used against us to lock us out of our own machines. We're no longer in the driver's seat. Our operating systems don't even operate the system anymore. Our free Linux systems are just the "user OS" in the manufacturer's unknowable amalgamation of silicon running proprietary firmware, just a little component to be sandboxed away from the real action.

Re: Hacked Nvidia 4090 GPU driver to enable P2P

#348

Post by wtallis » Sun, Apr 14, 2024, 5:05 PM UTC

papichulo2023 wrote: post #337

faeriechangling wrote: post #335

Earlier quoted context omitted.

Most consumer mobo's I see support this even if the setup isn't on the QVL. If a DDR5 motherboard support 4 sticks at all you can probably run 192gb on it so long as you update the BIOS firmware. The problem is running at rated speeds. AMD tends to be worse than Intel, and I hear people having to run anywhere between DDR5-3200 to DDR5-5200. You are better off running two sticks, because even with 2 sticks you really…

Hopefully Qualcom will free us of this 2 channels noghtmare.

I don't think it's realistic to pin your hopes on Qualcomm given that they're unlikely to care about supporting anything other than LPDDR with their laptop processors.

Re: Hacked Nvidia 4090 GPU driver to enable P2P

#349

Post by photonbeam » Sun, Apr 14, 2024, 6:12 PM UTC

chriskanan wrote: post #316

This is great news. As an academic, I'm aware of multiple labs that built boxes with 4090s, not realizing that Nvidia had impaired P2P communication among cards. It's one of the reasons I didn't buy 4090s, despite them being much more affordable for my work. It isn't nvlink, but Nvidia has mostly gotten rid of that except for their highest end cards. It is better than nothing. Late last year, I got quotes for machine…

How does cost compare with some of the GPU-cloud providers?

Re: Hacked Nvidia 4090 GPU driver to enable P2P

#350

Post by paulmd » Sun, Apr 14, 2024, 6:21 PM UTC

Sayrus wrote: post #315

axus wrote: post #92

Earlier quoted context omitted.

The price of 4090 may increase now, in theory locking out some features might have been a favor for some of the customers.

But it wouldn't if all cards supporting this were "unlocked" by default and thus the other "enterprise-grade" cards weren't that much more expensive. Of course that'd reduce profits by a lot.

it probably would - you saw exactly that outcome with mining.

for a lot of these demand bursts, demand is so high it cannot be sated even consuming 100% or 200% of typical GPU production.

cards like RX 6500XT that simply don't have the RAM to participate were less affected, but even then you've got enough cross-elasticity (demand from people being crowded out of other product segments) that tends to pump prices to 2-3x the "normal" clearance prices we see today. And yes, absolutely anything that can mine in any capacity will get pulled in during that sort of boom/bubble, not just "high-end"/"enterprise".