Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Hacked Nvidia 4090 GPU driver to enable P2P

github.com

Post reply on HN
Print view View original

141–150 of 365 posts

Re: Hacked Nvidia 4090 GPU driver to enable P2P

#141

Post by numpad0 »

andersa wrote: post #80

Incredible! I'd been wondering if this was possible. Now the only thing standing in the way of my 4x4090 rig for local LLMs is finding time to build it. With tensor parallelism, this will be both massively cheaper and faster for inference than a H100 SXM. I still don't understand why they went with 6 GPUs for the tinybox. Many things will only function well with 4 or 8 GPUs. It seems like the worst of both worlds now…

I was googling public NVIDIA SXM2 materials the other day, and it seemed SXM2/NVLink 2.0 just was a six-way system. NVIDIA SXM had updated to versions 3 and 4 since, and this isn't based on none of those anyway, but maybe there's something we don't know that make six-way reasonable.

Re: Hacked Nvidia 4090 GPU driver to enable P2P

#142

Post by andersa »

numpad0 wrote: post #141
andersa wrote: post #80

Incredible! I'd been wondering if this was possible. Now the only thing standing in the way of my 4x4090 rig for local LLMs is finding time to build it. With tensor parallelism, this will be both massively cheaper and faster for inference than a H100 SXM. I still don't understand why they went with 6 GPUs for the tinybox. Many things will only function well with 4 or 8 GPUs. It seems like the worst of both worlds now…

I was googling public NVIDIA SXM2 materials the other day, and it seemed SXM2/NVLink 2.0 just was a six-way system. NVIDIA SXM had updated to versions 3 and 4 since, and this isn't based on none of those anyway, but maybe there's something we don't know that make six-way reasonable.

It was probably just before running LLMs with tensor parallelism became interesting. There are plenty of other workloads that can be divided by 6 nicely, it's not an end-all thing.

Re: Hacked Nvidia 4090 GPU driver to enable P2P

#144

Post by bick_nyers »

andersa wrote: post #104
ShamelessC wrote: post #101

Earlier quoted context omitted.

> Many things will only function well with 4 or 8 GPUs What do you mean?

For example, if you want to run low latency multi-GPU inference with tensor parallelism in TensorRT-LLM, there is a requirement that the number of heads in the model is divisible by the number of GPUs. Most current published models are divisible by 4 and 8, but not 6.

Interesting... 1 Zen 4 EPYC CPU yields a maximum of 128 PCIE lanes so it wouldn't be possible to put 8 full fat GPUs on while maintaining some lanes for storage and networking. Same deal with Threadripper Pro.

Re: Hacked Nvidia 4090 GPU driver to enable P2P

#145

Post by andersa »

bick_nyers wrote: post #144
andersa wrote: post #104

Earlier quoted context omitted.

For example, if you want to run low latency multi-GPU inference with tensor parallelism in TensorRT-LLM, there is a requirement that the number of heads in the model is divisible by the number of GPUs. Most current published models are divisible by 4 and 8, but not 6.

Interesting... 1 Zen 4 EPYC CPU yields a maximum of 128 PCIE lanes so it wouldn't be possible to put 8 full fat GPUs on while maintaining some lanes for storage and networking. Same deal with Threadripper Pro.

It should be possible with onboard PCIe switches. You probably don't need the networking or storage to be all that fast while running the job, so it can dedicate almost all of the bandwidth to the GPU.

I don't know if there are boards that implement this, though, I'm only looking at systems with 4x GPUs currently. Even just plugging in a 5kW GPU server in my apartment would be a bit of a challenge. With 4x 4090, the max load would be below 3kW, so a single 240V plug can handle it no issue.

Re: Hacked Nvidia 4090 GPU driver to enable P2P

#147

Post by snakeyjake »

erikbye wrote: post #68
snakeyjake wrote: post #62

Earlier quoted context omitted.

I love the HN dystopian fantasies. They're simply adorable. They're like how jesusfreaks are constantly predicting the end times, with less mass suicide.

We already have export restrictions on cryptography. Of course there will be AI regulations.

You need to abandon your apocalyptic worldview keep up with the times my friend.

Encryption export controls have been systematically dismantled to the point that they're practically non-existent, especially over the last three years.

Pretty much the only encryption products you need permission to export are those specifically designed for integration into military communications networks, like Digital Subscriber Voice Terminals or Secure Terminal Equipment phones, everything else you file a form.

Many things have changed since the days when Windows 2000 shipped with a floppy disk containing strong encryption for use in certain markets.

https://archive.org/details/highencryptionfloppydisk

Re: Hacked Nvidia 4090 GPU driver to enable P2P

#148

Post by zoklet-enjoyer »

aftbit wrote: post #113
andy99 wrote: post #77

Earlier quoted context omitted.

Taxes, conscription and even pedestrian traffic rules make sense at least to some degree. Restricting "AI" because of what some uninformed politician imagines it to be is in a whole different league.

IMO it makes no sense to arrest someone and send them to jail for walking in the street not the sidewalk. Give them a ticket, make them pay a fine, sure, but force them to live in a cage with no access to communications, entertainment, or livelihood? Insane. Taxes may be necessary, though I can't help but feel that there must be a better way that we have not been smart enough to find yet. Conscription... is a fact of…

The US draft in the Vietnam war had nothing to do with the survival of the US

Re: Hacked Nvidia 4090 GPU driver to enable P2P

#149

Post by talldayo »

segfaultbuserr wrote: post #69
arghwhat wrote: post #21

Earlier quoted context omitted.

It does seem like an oversight, but there's nothing "suboptimal non-default options" about iteven if the implementation posted here seems somewhat hastily hacked together.

> but there's nothing "suboptimal non-default options" about it If "bypassing the official driver to invoke the underlying hardware feature directly through source code modification (and incompatibilities must be carefully worked around by turning off IOMMU and large BAR, since the feature was never officially supported)" does not count as "suboptimal non-default options", then I don't know what counts as "suboptimal…

> then I don't know what counts as "suboptimal non-default options".

Boy oh boy do I have a bridge to sell you: https://nouveau.freedesktop.org/

Re: Hacked Nvidia 4090 GPU driver to enable P2P

#150

Post by wmf »

spxneo wrote: post #139
llm_trw wrote: post #47

Earlier quoted context omitted.

Yes, the point here is that you do a direct write from one cards memory to the other using PCIe. In older NVidia cards this could be done through a faster link called NVLink but the hardware for that was ripped out of consumer grade cards and is only in data center grade cards now. Until this post it seemed like they had ripped all such functionality of their consumer cards, but it looks like you can still get it wor…

so whats stopping from somebody buying a ton of GPUs that are cheap and wiring it up via P2P like we saw with crypto mining

That's what this thread is about. Geohot is doing that.
Post reply on HN