Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Hacked Nvidia 4090 GPU driver to enable P2P

github.com

Post reply on HN
Print view View original

311–320 of 365 posts

Re: Hacked Nvidia 4090 GPU driver to enable P2P

#311

Post by spirobelv2 »

zoogeny wrote: post #171
jgpc wrote: post #93

Earlier quoted context omitted.

I agree. It is fascinating. When you observe his development process (btw, it is worth noting his generosity in sharing it like he does) he gets frequently stuck on random shallow problems which a perhaps more knowledgable engineer would find less difficult. It is frequent to see him writing really bad code, or even wrong code. The whole twitter chapter is a good example. Yet, himself, alone just iterating resilientl…

This matches my own take. I've tuned into a few of his streams and watched VODs on YouTube. I am consistently underwhelmed by his actual engineering abilities. He is that particular kind of engineer that constantly shits on other peoples code or on the general state of programming yet his actual code is often horrendous. He will literally call someone out for some code in Tinygrad that he has trouble with and then he…

link your github. want to see your raw intellectual power

Re: Hacked Nvidia 4090 GPU driver to enable P2P

#312

Post by treprinum »

LoganDark wrote: post #256
int_19h wrote: post #249

Earlier quoted context omitted.

It doesn't beat RTX 4090 when it comes to actual LLM inference speed. I bought a Mac Studio for local inference because it was the most convenient way to get something fast enough and with enough RAM to run even 155b models. It's great for that, but ultimately it's not magic - NVidia hardware still offers more FLOPS and faster RAM.

> It doesn't beat RTX 4090 when it comes to actual LLM inference speed Sure, whisper.cpp is not an LLM. The 4090 can't even do inference at all on anything over 24GB, while ASi can chug through it even if slightly slower. I wonder if with https://github.com/tinygrad/open-gpu-kernel-modules (the 4090 P2P patches) it might become a lot faster to split a too-large model across multiple 4090s and still outperform ASi (at…

Slightly slower in this case is like 10x. I have M3 Max with 128GB RAM, 4090 trashes it on anything under 24GB, then M3 Max trashes it on anything above 24GB, but it's like 10x slower at it than 4090 on <24GB.

Re: Hacked Nvidia 4090 GPU driver to enable P2P

#313

Post by FeepingCreature »

stale2002 wrote: post #285
FeepingCreature wrote: post #262

Earlier quoted context omitted.

Less than a month ago: https://arxiv.org/abs/2403.14380 "We found that participants who debated GPT-4 with access to their personal information had 81.7% (p And it's only gonna get better.

Yes, and I am sure that when people do a google search for "Good arguments in favor of X", that they are also sometimes convinced to be more in favor of X. Perhaps they would be even more convinced by the google search than if a person argued with them about it. That is still much different from "The AI mind controls people, hacks the nukes, and ends the world". Its that second part that is the the fantasy land situa…

If the only evidence for AI doom you will accept is actual AI doom, you are asking for evidence that by definition will be too late.

"Show me the AI mindcontrolling people!" AI mindcontrolling people is what we're trying to avoid seeing.

The trick is, in the world in which AI doom is in the future, what would you expect to see now that is different from the world in which AI doom is not in the future?

Re: Hacked Nvidia 4090 GPU driver to enable P2P

#314

Post by tinco »

WanderPanda wrote: post #283
georgehotz wrote: post #191

Earlier quoted context omitted.

tinygrad supports uneven splits. There's no fundamental reason for 4 or 8, and work should almost fully parallelize on any number of GPUs with good software. We chose 6 because we have 128 PCIe lanes, aka 8 16x ports. We use 1 for NVMe and 1 for networking, leaving 6 for GPUs to connect them in full fabric. If we used 4 GPUs, we'd be wasting PCIe, and if we used 8 there would be no room for external connectivity asid…

Did you at least front run the market and stocked up of 4090ies before this release? Also gamers are probably not too happy about these developments :D

4090's have consistently been around 2000 dollars. I don't think there's many gamers who would be affected by price fluctuations of the 4090 or even the 4080.

Re: Hacked Nvidia 4090 GPU driver to enable P2P

#315

Post by Sayrus »

axus wrote: post #92
rplnt wrote: post #89

Earlier quoted context omitted.

Then they couldn't charge different customers different amounts for the same HW. It's not a win for everyone.

The price of 4090 may increase now, in theory locking out some features might have been a favor for some of the customers.

But it wouldn't if all cards supporting this were "unlocked" by default and thus the other "enterprise-grade" cards weren't that much more expensive. Of course that'd reduce profits by a lot.

Re: Hacked Nvidia 4090 GPU driver to enable P2P

#316

Post by chriskanan »

This is great news. As an academic, I'm aware of multiple labs that built boxes with 4090s, not realizing that Nvidia had impaired P2P communication among cards. It's one of the reasons I didn't buy 4090s, despite them being much more affordable for my work. It isn't nvlink, but Nvidia has mostly gotten rid of that except for their highest end cards. It is better than nothing.

Late last year, I got quotes for machines with four nvlink H100s, but the lead time for delivery was 13 months. I could get the non-nvlink ones in just four months. For now, I've gone with four L40S cards to hold my lab over but supply chain issues and gigantic price increases are making it very hard for my lab to do it's work. That's not nearly enough to support 6 PhD students and a bunch of undergrads.

Things were a lot easier when I could just build machines with two GPUs each with Nvlink for $5K each and give one to each student to put under their desks, which is what I did back in 2015-2018 at my old university.

Re: Hacked Nvidia 4090 GPU driver to enable P2P

#317

Post by the8472 »

logicchains wrote: post #175
jprete wrote: post #98

Earlier quoted context omitted.

If AI is actually capable of fulfilling all the capabilities suggested by people who believe in the singularity, it has far more capacity for harm than nuclear weapons. I think most people who are strongly pro-AI/pro-acceleration - or, at any rate, not anti-AI - believe that either (A) there is no control problem (B) it will be solved (C) AI won't become independent and agentic (i.e. it won't face evolutionary pressu…

>If you hold the opposite opinions, then it makes perfect sense to push the brakes as hard as possible, which is why "govern compute" can make sense as an idea. The people pushing for "govern compute" are not pushing for "limit everyone's compute", they're pushing for "limit everyone's compute except us". Even if you believe there's going to be AGI, surely it's better to have distributed AGI than to have AGI only in…

Demonstrably false: https://twitter.com/ESYudkowsky/status/1772624785672954115

Re: Hacked Nvidia 4090 GPU driver to enable P2P

#318

Post by acka »

llm_trw wrote: post #47
ot1138 wrote: post #35

Earlier quoted context omitted.

Is this really efficient or practical? My understanding is that the latency required to copy memory from CPU or RAM to GPU negates any performance benefits (much less running over a network!)

Yes, the point here is that you do a direct write from one cards memory to the other using PCIe. In older NVidia cards this could be done through a faster link called NVLink but the hardware for that was ripped out of consumer grade cards and is only in data center grade cards now. Until this post it seemed like they had ripped all such functionality of their consumer cards, but it looks like you can still get it wor…

> In older NVidia cards this could be done through a faster link called NVLink but the hardware for that was ripped out of consumer grade cards and is only in data center grade cards now.

NVLink is still very much available in both RTX 3090 and A6000, both of which are still on the market. It was indeed removed from the RTX 40 series{0].

[0]: https://www.pugetsystems.com/labs/articles/nvidia-nvlink-202...

Re: Hacked Nvidia 4090 GPU driver to enable P2P

#319

Post by Aerroon »

segfaultbuserr wrote: post #189
logicchains wrote: post #175

Earlier quoted context omitted.

>If you hold the opposite opinions, then it makes perfect sense to push the brakes as hard as possible, which is why "govern compute" can make sense as an idea. The people pushing for "govern compute" are not pushing for "limit everyone's compute", they're pushing for "limit everyone's compute except us". Even if you believe there's going to be AGI, surely it's better to have distributed AGI than to have AGI only in…

> surely it's better to have distributed AGI than to have AGI only in the hands of the elites. The argument of doing so is the same as Nuclear Non-Proliferation - because of its great abuse potential, giving the technology to everyone only causes random bombings of cities instead of creating a system with checks and balances. I do not necessarily agree with it, but I found the reasoning is not groundless.

But the reason for nuclear non-proliferation is to hold onto power. Abuse potential is a great excuse, but it applies to everyone. Current nuclear states have demonstrated that they are willing to indirectly abuse them (you can't invade Russia, but Russia has no problem invading you as long as you aren't backed up by nukes).

Re: Hacked Nvidia 4090 GPU driver to enable P2P

#320

Post by CamperBob2 »

FeepingCreature wrote: post #263
CamperBob2 wrote: post #126

Earlier quoted context omitted.

Historically governments haven't needed computers or AI to do that. They've always managed just fine. Punched cards helped, though, I guess...

gestures at the human population graph wordlessly

Agent Smith smiles mirthlessly
Post reply on HN