Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Hacked Nvidia 4090 GPU driver to enable P2P

github.com

Post reply on HN
Print view View original

101–110 of 365 posts

Re: Hacked Nvidia 4090 GPU driver to enable P2P

#101

Post by ShamelessC »

andersa wrote: post #80

Incredible! I'd been wondering if this was possible. Now the only thing standing in the way of my 4x4090 rig for local LLMs is finding time to build it. With tensor parallelism, this will be both massively cheaper and faster for inference than a H100 SXM. I still don't understand why they went with 6 GPUs for the tinybox. Many things will only function well with 4 or 8 GPUs. It seems like the worst of both worlds now…

> Many things will only function well with 4 or 8 GPUs

What do you mean?

Re: Hacked Nvidia 4090 GPU driver to enable P2P

#103

Post by sparky_ »

llm_trw wrote: post #47
ot1138 wrote: post #35

Earlier quoted context omitted.

Is this really efficient or practical? My understanding is that the latency required to copy memory from CPU or RAM to GPU negates any performance benefits (much less running over a network!)

Yes, the point here is that you do a direct write from one cards memory to the other using PCIe. In older NVidia cards this could be done through a faster link called NVLink but the hardware for that was ripped out of consumer grade cards and is only in data center grade cards now. Until this post it seemed like they had ripped all such functionality of their consumer cards, but it looks like you can still get it wor…

I take it this is mostly useful for compute workloads, neural networks, LLM and the like -- not for actual graphics rendering?

Re: Hacked Nvidia 4090 GPU driver to enable P2P

#104

Post by andersa »

ShamelessC wrote: post #101
andersa wrote: post #80

Incredible! I'd been wondering if this was possible. Now the only thing standing in the way of my 4x4090 rig for local LLMs is finding time to build it. With tensor parallelism, this will be both massively cheaper and faster for inference than a H100 SXM. I still don't understand why they went with 6 GPUs for the tinybox. Many things will only function well with 4 or 8 GPUs. It seems like the worst of both worlds now…

> Many things will only function well with 4 or 8 GPUs What do you mean?

For example, if you want to run low latency multi-GPU inference with tensor parallelism in TensorRT-LLM, there is a requirement that the number of heads in the model is divisible by the number of GPUs. Most current published models are divisible by 4 and 8, but not 6.

Re: Hacked Nvidia 4090 GPU driver to enable P2P

#105

Post by segfaultbuserr »

ShamelessC wrote: post #101
andersa wrote: post #80

Incredible! I'd been wondering if this was possible. Now the only thing standing in the way of my 4x4090 rig for local LLMs is finding time to build it. With tensor parallelism, this will be both massively cheaper and faster for inference than a H100 SXM. I still don't understand why they went with 6 GPUs for the tinybox. Many things will only function well with 4 or 8 GPUs. It seems like the worst of both worlds now…

> Many things will only function well with 4 or 8 GPUs What do you mean?

It's more difficult to split your work across 6 GPUs evenly, and easier when you have 4 or 8 GPUs. The latter setups have powers of 2, which for example, can evenly divide a 2D or 3D grid, but 6 GPUs are awkward to program. Thus, the OP argues that a 6-GPU setup is highly suboptimal for many existing applications and there's no point to pay more for the extra 2.

Re: Hacked Nvidia 4090 GPU driver to enable P2P

#106

Post by magicalhippo »

HeatrayEnjoyer wrote: post #55
magicalhippo wrote: post #36

Earlier quoted context omitted.

I don't know about this particular scenario, but typically fuses are small wires or resistors that are overloaded so they irreversibly break the connection. Hence the name. Either done during manufacture or as a one-time programming[1][2]. Though sometimes reprogrammable configuration bits are sometimes also called fuse bits. The Atmega328P of Arduino fame uses flash[3] for its "fuses". [1]: https://www.nxp.com/docs/…

Wires, flash, and resistors can be replaced

These are part of the chip, thus microscopic and very inaccessible.

There are some good images here[1] of various such fuses, both pristine and blown. Here's[2] a more detailed writeup examining one type.

It's not something you fix with a soldering iron.

[1]: https://semiengineering.com/the-benefits-of-antifuse-otp/

[2]: https://www.eetimes.com/a-look-at-metal-efuses/

Re: Hacked Nvidia 4090 GPU driver to enable P2P

#107

Post by andersa »

corn13read2 wrote: post #94
andersa wrote: post #80

Incredible! I'd been wondering if this was possible. Now the only thing standing in the way of my 4x4090 rig for local LLMs is finding time to build it. With tensor parallelism, this will be both massively cheaper and faster for inference than a H100 SXM. I still don't understand why they went with 6 GPUs for the tinybox. Many things will only function well with 4 or 8 GPUs. It seems like the worst of both worlds now…

A macbook is cheaper though

Sure, it's also at least an order of magnitude slower in practice, compared to 4x 4090 running at full speed. We're looking at 10 times the memory bandwidth and much greater compute.

Re: Hacked Nvidia 4090 GPU driver to enable P2P

#110

Post by mikepurvis »

jaimehrubiks wrote: post #82
rfoo wrote: post #24

Glad to see that geohot is back being geohot, first by dropping a local DoS for AMD cards, then this. Much more interesting :p

Is this the same guy that hacked the PS3?

Yes, but he spent several years in self-driving cars (https://comma.ai), which while interesting is also a space that a lot of players are in, so it's not the same as seeing him back to doing stuff that's a little more out there, especially as pertains to IP.
Post reply on HN