Live data from Hacker News

Hacked Nvidia 4090 GPU driver to enable P2P

github.com

351–360 of 365 posts

Re: Hacked Nvidia 4090 GPU driver to enable P2P

#351

This is great news. As an academic, I'm aware of multiple labs that built boxes with 4090s, not realizing that Nvidia had impaired P2P communication among cards. It's one of the reasons I didn't buy 4090s, despite them being much more affordable for my work. It isn't nvlink, but Nvidia has mostly gotten rid of that except for their highest end cards. It is better than nothing. Late last year, I got quotes for machine…

How does cost compare with some of the GPU-cloud providers?

Not op, but I found this benchmark of whisper large-v3 interesting [1]. It includes the cloud provider's pricing per gpu, so you can directly calculate break-even timing.

Of course, if you use different models, training, fine tuning etc. the benchmarks will differ depending on ram, support of fp8 etc.

[1] https://blog.salad.com/whisper-large-v3/

Re: Hacked Nvidia 4090 GPU driver to enable P2P

#352

Earlier quoted context omitted.

tinygrad supports uneven splits. There's no fundamental reason for 4 or 8, and work should almost fully parallelize on any number of GPUs with good software. We chose 6 because we have 128 PCIe lanes, aka 8 16x ports. We use 1 for NVMe and 1 for networking, leaving 6 for GPUs to connect them in full fabric. If we used 4 GPUs, we'd be wasting PCIe, and if we used 8 there would be no room for external connectivity asid…

Did you at least front run the market and stocked up of 4090ies before this release? Also gamers are probably not too happy about these developments :D

Gamers have a TON of really good really affordable options. But you kind of need 24gb min unless you're using heavy quantization. So 3090 and 4090's are what local llm people are building with (mostly 3090's as you can get then for about $700, and they're dang good)

Re: Hacked Nvidia 4090 GPU driver to enable P2P

#353
post #86

Earlier quoted context omitted.

Sort of. An imperfect analogy: a small neighborhood of ~15 houses is under construction. Normally it might have a 200kva transformer sitting at the corner, which provides appropriate power from the grid. But there is a transformer shortage, so the contractor installs a commercial grade 1250kva transformer. It can power many more houses than required, so it's operating way under capacity. One day, a resident decides h…

And then because this residential neighborhood now has commercial grade power, the other lots that were going to have residential houses built on them instead get combined into a factory, and the people who want to buy new houses in town have to pay more since residential supply was cut in half.

This represents pretty well how gamers (residential buyers) are going to feel when the next generation of consumer cards are scooped up for AI.

Re: Hacked Nvidia 4090 GPU driver to enable P2P

#354
so basically rtx 4090 x6 = 144 GB ram which would cost $15996 = $9594 ( only the nvidia 4090s) and currently the tiny box gives *TinyBox* > GPU RAM | 144 GB > Price | $15,000 $25,000 Nvidia 4090x6

> GPU RAM | 144 GB > Price | $9594

so a * 36.04%* decrease in price from team red tinybox ( $15k) and *61.624% *decrease in price from the team green tinybox ( $25k)

Re: Hacked Nvidia 4090 GPU driver to enable P2P

#355

Earlier quoted context omitted.

This matches my own take. I've tuned into a few of his streams and watched VODs on YouTube. I am consistently underwhelmed by his actual engineering abilities. He is that particular kind of engineer that constantly shits on other peoples code or on the general state of programming yet his actual code is often horrendous. He will literally call someone out for some code in Tinygrad that he has trouble with and then he…

link your github. want to see your raw intellectual power

I know what I said about lacking raw intellectual power probably feels like a personal attack rather than a description. However, that comment is in comparison to guys like Peter Norvig or Donald Knuth, not random Hacker News mid-wits like myself.

I had a younger cousin who wanted to start a career in software engineering. He asked me, assuming my years of experience had some merit, what programming languages to learn, what code editor to use, what platforms and frameworks to study. I told him the most important thing he could do is to be persistent. The computer will constantly humble you. Your coworkers will constantly try to rail-road you into solutions that are sub-optimal. You have to be resilient and keep going no matter what, you can't ever give up.

I think it is fair to say that George excels at what I consider to be the most important aspect of programming. And if he could manage to stop disparaging others in his streams, suggesting that everyone else is stupid and that the code they write is rotten, I could very easily look over the fact that he is frequently careless and hasty.

Re: Hacked Nvidia 4090 GPU driver to enable P2P

#356

Earlier quoted context omitted.

Are you on drugs or is your reading comprehension that poor? 1) I did not state a world view; I simply noted that restrictions for software do exist, and will for AI, as well. As the link from the other commenter show, they do in fact already exist. 2) Look up the definition of "apocalyptic", software restrictions are not within its bounds. 3) How the restrictions are enforced were not a subject in my comment. 4) We'…

I'm high on life, old chum!

You enjoy, then :)

Re: Hacked Nvidia 4090 GPU driver to enable P2P

#357

Earlier quoted context omitted.

Is it possible a similar patch would work for P2P on 3090s? btw, I found a Gigabyte board on Taobao that is unlisted on their site: MZF2-AC0, costs $900. 2 socket Epyc and 10 PCIE slots, may be of interest. A case that should fit, with 2x 2000W Great Wall PSUs and PDU is 4050 RMB ( https://www.toploong.com/en/4GPU-server-case/644.html ). You still need blower GPUs.

It should if your 3090s have Resizable BAR support in the VBIOS. AFAIK most card manufacturers released BIOS updates enabling this. Re: 3090 NVLink, that only allows pairs of cards to be connected. PCIe allows full fabric switch of many cards.

Thanks for the amazing work! I tried the driver with some 3090s (all of which show the 32G line with lspci -s 01:00.0 -v) and while torch says I have p2p access, I can't get it to work with anything as I get illegal memory access errors.

Re: Hacked Nvidia 4090 GPU driver to enable P2P

#358

Earlier quoted context omitted.

> bypassing the official driverto The driver is not bypasses. This is a patch to the official open-source kernel-driver where the feature is added, which is how all upstream Linux driver development is done. > to invoke the underlying hardware feature directly Accessing hardware features directly is pretty much the sole job of a driver, and the only thing "bypassed" is some abstractions internal to the driver. Just m…

> The driver is not bypasses. This is a patch to the official open-source kernel-driver where the feature is added, which is how all upstream Linux driver development is done. [source code modification] is a weird way to describe software engineering. Making the code available for further development is kind of the whole point of open source. My previous comment was written with an unspoken assumption: Hardware drive…

> For ordinary free-and-open-source software, the source code availability largely guarantees community control. However, the same often does not apply to drivers. Even with source code availability, they're often written by vendors using NDAed information and in-house expertise about the underlying hardware design. As a result, drivers remain under a vendor's tight control.

This is not true for any upstream Linux kernel driver. They are fully open source, fully controlled by the community and not subject to any NDAs to work on. The vendor can only exert control in the form of reviews and open maintainership. This is the case for both AMD and Intel GPU drivers.

For numbers, Arch Linux bundles some ~7.5k loadable kernel drivers (which is a subset of all available, but AUR only has 8 out-of-tree drivers, out of which two are proprietary and 2-3 are community-led. "vendor-controlled" drivers are an extreme outlier on Linux.

While not true for nvidia's proprietary, closed-source driver stack, the linked open source nvidia (kernel) driver is nvidia's work-in-progress driver to be upstreamed in exactly this fashion.

> by watching what happens when you send the commands to hardware to invoke a feature unexposed by the HAL. Thus, it was found by reverse-engineering

It is true that documentation was lacking, but there was no reverse-engineering here, just standard (albeit, hacked up) driver development. "The HAL" means nothing, as it's just an random abstraction within the driver.

I used to work on proprietary network drivers for high-performance, FPGA-based NICs, and apart from it being more annoying to debug it's really no different than coding on anything else. Unless you're bringing up new IP blocks, it's mostly you and the code, not specs.

It would have been reverse-engineering if the registers and blocks weren't in the driver in the first place, but in this case all the parts were there to bring up a completely standard feature. What he did was use existing driver code to ask for what he wanted.

Not saying that it wasn't significant effort, but unless you consider "reading code you did not write" reverse-engineering (in which case all coding is reverse-engineering), then it has nothing to do with reverse-engineering, or bypassing anything. Considering this reverse-engineering is also a bit of a disservice to those that actually reverse-engineered GPUs and wrote open-source GPU drivers for them, like panfrost and recently the Apple silicon drivers.

Apart from the fact that the feature is hacked and not properly wired up (as evident by the IOMMU issue and abstraction break), the implementation flow is done exactly the same as when volunteers contribute fixes or features to other, complex open-source upstream GPU drivers, like AMDGPU. Not by reverse-engineering, nor by signing NDAs, but by reading and writing code and debugging your hardware. Stuff is just always harder in kernel mode.

> I meant "turning off (IOMMU && large BAR)". Disabling large BAR in order to use PCIe P2P is a suboptimal configuration.

The requirement is large BAR on, not off - the phrasing is a bit poor on the page, but they're trying to say that you need large bar support, and that you need IOMMU off, not that you need large bar and IOMMU off.

Re: Hacked Nvidia 4090 GPU driver to enable P2P

#360

Earlier quoted context omitted.

>If you hold the opposite opinions, then it makes perfect sense to push the brakes as hard as possible, which is why "govern compute" can make sense as an idea. The people pushing for "govern compute" are not pushing for "limit everyone's compute", they're pushing for "limit everyone's compute except us". Even if you believe there's going to be AGI, surely it's better to have distributed AGI than to have AGI only in…

Demonstrably false: https://twitter.com/ESYudkowsky/status/1772624785672954115

Do they also take this position about biological intelligence? Because humans most certainly have an alignment problem too.
Post reply on HN