Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Hacked Nvidia 4090 GPU driver to enable P2P

github.com

Post reply on HN
Print view View original

291–300 of 365 posts

Re: Hacked Nvidia 4090 GPU driver to enable P2P

#291

Post by Dylan16807 »

stavros wrote: post #281
cesarb wrote: post #258

Earlier quoted context omitted.

Except that this "lots of power" option does not exist. What limits the amount of power used is the circuit breakers and fuses on the panel, which protect the wiring against overheating by tripping when too much power is being used (or when there's a short circuit). The resident in this analogy would need to ensure that not only the transformer, but also the wiring leading to the transformer, can handle the higher cu…

There are analogies, and then there's this.

They're pointing out how the analogy doesn't work, so it's fine.

Nobody's taking more than their share of any resources when they enable this feature.

Re: Hacked Nvidia 4090 GPU driver to enable P2P

#292

Post by [deleted] »

numpad0 wrote: post #230
spxneo wrote: post #139

Earlier quoted context omitted.

so whats stopping from somebody buying a ton of GPUs that are cheap and wiring it up via P2P like we saw with crypto mining

PCIe P2P still has to go up to a central hub thing and back because PCIe is not a bus. That central hub thing is made by very few players(most famously PLX Technologies) and it costs a lot.

[deleted]

Re: Hacked Nvidia 4090 GPU driver to enable P2P

#293

Post by publicmail »

wmf wrote: post #155
CamperBob2 wrote: post #122

Earlier quoted context omitted.

The correct term, and the one most people would have used in the past, is "bus mastering."

PCIe isn't a bus and it doesn't really have a concept of mastering. All PCI DMA was based on bus mastering but P2P DMA is trickier than normal DMA.

I consider it bus mastering when the endpoints initiate the transactions

Re: Hacked Nvidia 4090 GPU driver to enable P2P

#294

Post by publicmail »

ot1138 wrote: post #35
__alexs wrote: post #34

Earlier quoted context omitted.

It means you can send data from the memory of 1 GPU to another GPU without going via RAM. https://xilinx.github.io/XRT/master/html/p2p.html

Is this really efficient or practical? My understanding is that the latency required to copy memory from CPU or RAM to GPU negates any performance benefits (much less running over a network!)

PCIe busses are like a tree with “hubs” (really switches).

Imagine you have a PC with a PCIe x16 interface which is attached to a PCIe switch that has four x16 downstream ports, each attached to a GPU. Those GPUs are capable of moving data in and out of their PCIe interfaces at full speed.

If you wanted to transfer data from GPU0 and 1 to GPU2 and 3, you have basically 2 options:

- Have GPU0 and 1 move their data to CPU DRAM, then have GPU2 and 3 fetch it

- Have GPU0 and 1 write their data directly to GPU2 and 3 through the switch they’re connected to without ever going up to the CPU at all

In this case, option 2 is better both because it avoids the extra copy to CPU DRAM and also because it avoids the bottleneck of two GPUs trying to push x16 worth of data up through the CPUs single x16 port. This is known as peer to peer.

There are some other scenarios where the data still must go up to the CPU port and back due to ACS, and this is still technically P2P, but doesn’t avoid the bottleneck like routing through the switch would.

Re: Hacked Nvidia 4090 GPU driver to enable P2P

#295

Post by gorkish »

jgpc wrote: post #93
ivanjermakov wrote: post #54

I was always fascinated by George Hotz's hacking abilities. Inspired me a lot for my personal projects.

I agree. It is fascinating. When you observe his development process (btw, it is worth noting his generosity in sharing it like he does) he gets frequently stuck on random shallow problems which a perhaps more knowledgable engineer would find less difficult. It is frequent to see him writing really bad code, or even wrong code. The whole twitter chapter is a good example. Yet, himself, alone just iterating resilientl…

If a stopped clock is right twice a day, relentlessly winding a clock forward will make it right quite frequently. That is geohot.

Re: Hacked Nvidia 4090 GPU driver to enable P2P

#296

Post by TylerE »

dheera wrote: post #220
numpad0 wrote: post #141

Earlier quoted context omitted.

I was googling public NVIDIA SXM2 materials the other day, and it seemed SXM2/NVLink 2.0 just was a six-way system. NVIDIA SXM had updated to versions 3 and 4 since, and this isn't based on none of those anyway, but maybe there's something we don't know that make six-way reasonable.

What is a six-way system?

oid school way of saying core (or in this case GPU), basically.

Re: Hacked Nvidia 4090 GPU driver to enable P2P

#297

Post by davidzweig »

davidzweig wrote: post #201
georgehotz wrote: post #191

Earlier quoted context omitted.

tinygrad supports uneven splits. There's no fundamental reason for 4 or 8, and work should almost fully parallelize on any number of GPUs with good software. We chose 6 because we have 128 PCIe lanes, aka 8 16x ports. We use 1 for NVMe and 1 for networking, leaving 6 for GPUs to connect them in full fabric. If we used 4 GPUs, we'd be wasting PCIe, and if we used 8 there would be no room for external connectivity asid…

Is it possible a similar patch would work for P2P on 3090s? btw, I found a Gigabyte board on Taobao that is unlisted on their site: MZF2-AC0, costs $900. 2 socket Epyc and 10 PCIE slots, may be of interest. A case that should fit, with 2x 2000W Great Wall PSUs and PDU is 4050 RMB ( https://www.toploong.com/en/4GPU-server-case/644.html ). You still need blower GPUs.

Update, I checked with the case company, toploong, they say that board is a 5mm too big or so for the case.

Re: Hacked Nvidia 4090 GPU driver to enable P2P

#298

Post by thangngoc89 »

wtallis wrote: post #211
thangngoc89 wrote: post #120

Earlier quoted context omitted.

training on MPS backend is suboptimal and really slow.

Do people do training on systems this small, or just inference? I could see maybe doing a little bit of fine-tuning, but certainly not from-scratch training.

Yep. Price/performance of multiple 4090s system are way better than the professional cards (Axxx). Also deep learning outside of LLM has many different usage.

Re: Hacked Nvidia 4090 GPU driver to enable P2P

#299

Post by ctrw »

zoogeny wrote: post #171
jgpc wrote: post #93

Earlier quoted context omitted.

I agree. It is fascinating. When you observe his development process (btw, it is worth noting his generosity in sharing it like he does) he gets frequently stuck on random shallow problems which a perhaps more knowledgable engineer would find less difficult. It is frequent to see him writing really bad code, or even wrong code. The whole twitter chapter is a good example. Yet, himself, alone just iterating resilientl…

This matches my own take. I've tuned into a few of his streams and watched VODs on YouTube. I am consistently underwhelmed by his actual engineering abilities. He is that particular kind of engineer that constantly shits on other peoples code or on the general state of programming yet his actual code is often horrendous. He will literally call someone out for some code in Tinygrad that he has trouble with and then he…

There are developers who have breadth and developers who have depth. He is very much on the breadth end of the spectrum. It isn't lack of intelligence but lack of deep knowledge of esoteric fields you will use once a decade.

That said I find it a bit astonishing how little Ai he uses on his streams. I convert all the documentation I need into a rag system that I query stupid questions against.

Re: Hacked Nvidia 4090 GPU driver to enable P2P

#300

Post by georgehotz »

samtheprogram wrote: post #290
pixelpoet wrote: post #266

Earlier quoted context omitted.

Ah yes, nothing like a bit of hypocrisy to make a point. It's okay though, as long as it's people we don't agree with, defrauding them is fine.

The website literally stated it was not for speculation, they didn't want the price to go up, and there were multiple ways to get some for free. If people were reckless, greedy, and/or lazy because of the crypto hype and got "defrauded" without doing any amount of due diligence -- that's kinda the point.

I actually lost about $5k on cheapETH running servers. Nobody was "defrauded", I think these people don't understand how forks work. It's a precursor to the modern L2 stuff, I did this while writing the first version of Optimism's fraud prover. https://github.com/ethereum-optimism/cannon

I suspect most of the people who bring this up don't like me for other reasons, but with this they think they have something to latch on to. Doesn't matter that it isn't true and there wasn't a scam, they aren't going to look into it since it agrees with their narrative.

Post reply on HN