Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Hacked Nvidia 4090 GPU driver to enable P2P

github.com

Post reply on HN
Print view View original

191–200 of 365 posts

Re: Hacked Nvidia 4090 GPU driver to enable P2P

#191

Post by georgehotz »

andersa wrote: post #80

Incredible! I'd been wondering if this was possible. Now the only thing standing in the way of my 4x4090 rig for local LLMs is finding time to build it. With tensor parallelism, this will be both massively cheaper and faster for inference than a H100 SXM. I still don't understand why they went with 6 GPUs for the tinybox. Many things will only function well with 4 or 8 GPUs. It seems like the worst of both worlds now…

tinygrad supports uneven splits. There's no fundamental reason for 4 or 8, and work should almost fully parallelize on any number of GPUs with good software.

We chose 6 because we have 128 PCIe lanes, aka 8 16x ports. We use 1 for NVMe and 1 for networking, leaving 6 for GPUs to connect them in full fabric. If we used 4 GPUs, we'd be wasting PCIe, and if we used 8 there would be no room for external connectivity aside from a few USB3 ports.

Re: Hacked Nvidia 4090 GPU driver to enable P2P

#192

Post by pixl97 »

raxxorraxor wrote: post #44
the8472 wrote: post #28

Earlier quoted context omitted.

... after regulation, court orders and fines have failed. Which under the premise that AGI is an existential threat would be far more reasonable than many other reasons for raids. If the premise is wrong we won't need it. If society coordinates to not do the dangerous thing we won't need it. The argument is that only in the case where we find ourselves in the situation where other measures have failed such uses of fo…

To be honest, I see summoning the threat of AGI to pose an existential threat to be on the level with lizard people on the moon. Great for sci-fi, bad distraction for policy making and addressing real problems. The real war, if there is one, is about owning data and collecting data. And surprisingly many people fall for distractions while their LLM fails at basic math. Because it is a language model of course...

> I see summoning the threat of AGI to pose an existential threat to be on the level with lizard people on the moon.

I mean to every other lifeform on the plant YOU are the AGI existential threat. You, and I mean homosapiens by that, have taken over the planet and have either enslaved and are breeding any other animals for food, or are driving them to extinction. In this light bringing another potential apex predator on to the scene seems rash.

>fall for distractions while their LLM fails at basic math

Correct, if we already had AGI/ASI this discussion would be moot because we'd already be in a world of trouble. The entire point is to slow stuff down before we have a major "oopsie whoopsie we can't take that back" issue with advanced AI, and the best time to set the rules is now.

Re: Hacked Nvidia 4090 GPU driver to enable P2P

#193

Post by cavisne »

LoganDark wrote: post #184
tgtweak wrote: post #99

Earlier quoted context omitted.

The extra $3k you'd spend on a quad-4090 rig vs the top mbp... ignoring the fact you can't put the two on even ground for versatility (very few libraries are adapted to apple silicone let alone optimized). Very few people that would consider an H100/A100/A800 are going to be cross-shopping a macbook pro for their workloads.

> very few libraries are adapted to apple silicone let alone optimized This is a joke, right? Have you been anywhere in the LLM ecosystem for the past year or so? I'm constantly hearing about new ways in which ASi outperforms traditional platforms, and new projects that are optimized for ASi. Such as, for instance, llama.cpp.

Nothing compared to Nvidia though. The FLOPS and memory bandwidth is simply not there.

Re: Hacked Nvidia 4090 GPU driver to enable P2P

#194

Post by [deleted] »

segfaultbuserr wrote: post #189
logicchains wrote: post #175

Earlier quoted context omitted.

>If you hold the opposite opinions, then it makes perfect sense to push the brakes as hard as possible, which is why "govern compute" can make sense as an idea. The people pushing for "govern compute" are not pushing for "limit everyone's compute", they're pushing for "limit everyone's compute except us". Even if you believe there's going to be AGI, surely it's better to have distributed AGI than to have AGI only in…

> surely it's better to have distributed AGI than to have AGI only in the hands of the elites. The argument of doing so is the same as Nuclear Non-Proliferation - because of its great abuse potential, giving the technology to everyone only causes random bombings of cities instead of creating a system with checks and balances. I do not necessarily agree with it, but I found the reasoning is not groundless.

[deleted]

Re: Hacked Nvidia 4090 GPU driver to enable P2P

#195

Post by pixl97 »

eek2121 wrote: post #112
the8472 wrote: post #28

Earlier quoted context omitted.

... after regulation, court orders and fines have failed. Which under the premise that AGI is an existential threat would be far more reasonable than many other reasons for raids. If the premise is wrong we won't need it. If society coordinates to not do the dangerous thing we won't need it. The argument is that only in the case where we find ourselves in the situation where other measures have failed such uses of fo…

Just my (probably unpopular) opinion: True AI (what they are now calling AGI) may never exist. Even the AI models of today aren't far removed from the 'chatbots' of yesterday (more like an evolution rather than revolution)... ...for true AI to exist, it would need to be self aware. I don't see that happening in our lifetimes when we don't even know how our own brains work. (There is sooo much we don't know about the…

Why is "Want" even part of your equation.

Bacteria doesn't "want" anything in the sense of active thinking like you do, and yet will render you dead quickly and efficiently while spreading at a near exponential rate. No self awareness necessary.

You keep drawing little circles based on your understanding of the world and going "it's inside this circle, therefore I don't need to worry about it", while ignoring 'semi-smart' optimization systems that can lead to dangerous outcomes.

>I am old enough to have gone through numerous bubbles,

And evidently not old enough to pay attention to the things that did pan out. But hey, those cellphone and that internet thing was just a fad right. We'll go back to land lines at any time now.

Re: Hacked Nvidia 4090 GPU driver to enable P2P

#196

Post by doctorpangloss »

georgehotz wrote: post #191
andersa wrote: post #80

Incredible! I'd been wondering if this was possible. Now the only thing standing in the way of my 4x4090 rig for local LLMs is finding time to build it. With tensor parallelism, this will be both massively cheaper and faster for inference than a H100 SXM. I still don't understand why they went with 6 GPUs for the tinybox. Many things will only function well with 4 or 8 GPUs. It seems like the worst of both worlds now…

tinygrad supports uneven splits. There's no fundamental reason for 4 or 8, and work should almost fully parallelize on any number of GPUs with good software. We chose 6 because we have 128 PCIe lanes, aka 8 16x ports. We use 1 for NVMe and 1 for networking, leaving 6 for GPUs to connect them in full fabric. If we used 4 GPUs, we'd be wasting PCIe, and if we used 8 there would be no room for external connectivity asid…

Have you compared 3x 3090-3090 pairs over NVLink?

IMO the most painful thing is that since these hardware configurations are esoteric, there is no software that detects them and moves things around "automatically." Regardless of what people thing device_map="auto" does, and anyway, Hugging Face's transformers/diffusers are all over the place.

Re: Hacked Nvidia 4090 GPU driver to enable P2P

#197

Post by pixl97 »

m3kw9 wrote: post #176
mvkel wrote: post #86

Earlier quoted context omitted.

Sort of. An imperfect analogy: a small neighborhood of ~15 houses is under construction. Normally it might have a 200kva transformer sitting at the corner, which provides appropriate power from the grid. But there is a transformer shortage, so the contractor installs a commercial grade 1250kva transformer. It can power many more houses than required, so it's operating way under capacity. One day, a resident decides h…

Where is the hack in this analogy

Taking off the users panel on the side of their house and flipping it to 'lots of power' when that option had previously been covered up by the panel interface.

Re: Hacked Nvidia 4090 GPU driver to enable P2P

#198

Post by [deleted] »

pixelpoet wrote: post #185
dji4321234 wrote: post #169

Earlier quoted context omitted.

He has a very checkered history with "hacking" things. He tends to build heavily on the work of others, then use it to shamelessly self-promote, often to the massive detriment of the original authors. His PS3 work was based almost completely on a presentation given by fail0verflow at CCC. His subsequent self-promotion grandstanding world tour led to Sony suing both him and fail0verflow, an outcome they were specifica…

There was also that CheapEth crypto scam he tried to pull off.

[deleted]

Re: Hacked Nvidia 4090 GPU driver to enable P2P

#199

Post by [deleted] »

delfinom wrote: post #178
dji4321234 wrote: post #169

Earlier quoted context omitted.

He has a very checkered history with "hacking" things. He tends to build heavily on the work of others, then use it to shamelessly self-promote, often to the massive detriment of the original authors. His PS3 work was based almost completely on a presentation given by fail0verflow at CCC. His subsequent self-promotion grandstanding world tour led to Sony suing both him and fail0verflow, an outcome they were specifica…

[flagged]

[deleted]

Re: Hacked Nvidia 4090 GPU driver to enable P2P

#200

Post by bick_nyers »

andersa wrote: post #145
bick_nyers wrote: post #144

Earlier quoted context omitted.

Interesting... 1 Zen 4 EPYC CPU yields a maximum of 128 PCIE lanes so it wouldn't be possible to put 8 full fat GPUs on while maintaining some lanes for storage and networking. Same deal with Threadripper Pro.

It should be possible with onboard PCIe switches. You probably don't need the networking or storage to be all that fast while running the job, so it can dedicate almost all of the bandwidth to the GPU. I don't know if there are boards that implement this, though, I'm only looking at systems with 4x GPUs currently. Even just plugging in a 5kW GPU server in my apartment would be a bit of a challenge. With 4x 4090, the…

I've seen it done with a PLX Multiplexer as well, but they add quite a bit of cost:

https://c-payne.com/products/pcie-gen4-switch-backplane-4-x1...

Not sure if there exists an 8-way PCIE Gen 5 Multiplexer that doesn't cost ludicrous amounts of cash. Ludicrous being a highly subjective and relative term of course.

Post reply on HN