Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Hacked Nvidia 4090 GPU driver to enable P2P

github.com

Post reply on HN
Print view View original

281–290 of 365 posts

Re: Hacked Nvidia 4090 GPU driver to enable P2P

#281

Post by stavros »

cesarb wrote: post #258
pixl97 wrote: post #197

Earlier quoted context omitted.

Taking off the users panel on the side of their house and flipping it to 'lots of power' when that option had previously been covered up by the panel interface.

Except that this "lots of power" option does not exist. What limits the amount of power used is the circuit breakers and fuses on the panel, which protect the wiring against overheating by tripping when too much power is being used (or when there's a short circuit). The resident in this analogy would need to ensure that not only the transformer, but also the wiring leading to the transformer, can handle the higher cu…

There are analogies, and then there's this.

Re: Hacked Nvidia 4090 GPU driver to enable P2P

#282

Post by pests »

namibj wrote: post #224
bick_nyers wrote: post #200

Earlier quoted context omitted.

I've seen it done with a PLX Multiplexer as well, but they add quite a bit of cost: https://c-payne.com/products/pcie-gen4-switch-backplane-4-x1... Not sure if there exists an 8-way PCIE Gen 5 Multiplexer that doesn't cost ludicrous amounts of cash. Ludicrous being a highly subjective and relative term of course.

98 lanes of PCIe 4.0 fabric switch as just the chip (to solder onto a motherboard/backplane) costs 850$ (PEX88096). You could for example take 2 x16 GPUs, pass then through (2 2 16=64 lanes), and have 2 x16 that bifurcate to at least x4 (might even be x2, I didn't find that part of the docs just now) for anything you want, plus 2 x1 for minor stuff. They do claim to have no problems being connected up into a switchin…

I really like the information about how the cloud providers do their multiplexing, thanks. There was some tech posted here a few month ago that was similar I found very interesting - plug all devices, ram, hard drives, and CPUS into a larger fabric and a way to spin up "servers" of any size from the pool of resources... wish I could remember the name now.

nit: HN formatting messed up your math in the second sentence, I believe you italicized on accident using * for equations.

Re: Hacked Nvidia 4090 GPU driver to enable P2P

#283

Post by WanderPanda »

georgehotz wrote: post #191
andersa wrote: post #80

Incredible! I'd been wondering if this was possible. Now the only thing standing in the way of my 4x4090 rig for local LLMs is finding time to build it. With tensor parallelism, this will be both massively cheaper and faster for inference than a H100 SXM. I still don't understand why they went with 6 GPUs for the tinybox. Many things will only function well with 4 or 8 GPUs. It seems like the worst of both worlds now…

tinygrad supports uneven splits. There's no fundamental reason for 4 or 8, and work should almost fully parallelize on any number of GPUs with good software. We chose 6 because we have 128 PCIe lanes, aka 8 16x ports. We use 1 for NVMe and 1 for networking, leaving 6 for GPUs to connect them in full fabric. If we used 4 GPUs, we'd be wasting PCIe, and if we used 8 there would be no room for external connectivity asid…

Did you at least front run the market and stocked up of 4090ies before this release? Also gamers are probably not too happy about these developments :D

Re: Hacked Nvidia 4090 GPU driver to enable P2P

#284

Post by waldrews »

Would this approach be possible to extend downmarket, to older consumer cards? For a lot of LLM use cases we're constrained by memory and can tolerate lower compute speeds so long as there's no swapping. ELI5, what would prevent a hundred 1060-level cards from being used together?

Re: Hacked Nvidia 4090 GPU driver to enable P2P

#285

Post by stale2002 »

FeepingCreature wrote: post #262
stale2002 wrote: post #123

Earlier quoted context omitted.

AI mind control abilities are also on the level of an extraordinary claim, that requires extraordinary evidence. It's on the level of "we better regulate wooden sticks so Voldemort doesn't use the imperious curse on us!". That's how I treat such claims. I treat them the same as someone literally talking about magic from Harry potter. There isn't nothing that would make me believe that. But it requires actual evidence…

Less than a month ago: https://arxiv.org/abs/2403.14380 "We found that participants who debated GPT-4 with access to their personal information had 81.7% (p And it's only gonna get better.

Yes, and I am sure that when people do a google search for "Good arguments in favor of X", that they are also sometimes convinced to be more in favor of X.

Perhaps they would be even more convinced by the google search than if a person argued with them about it.

That is still much different from "The AI mind controls people, hacks the nukes, and ends the world".

Its that second part that is the the fantasy land situation that requires extraordinary evidence.

But, this is how conversations about doomsday AI always go. People say "Well isn't AI kinda good at this extremely vague thing Y, sometimes? Imagine if AI was infinitely good at Y! That means that by extrapolation, the world ends!".

And that covers basically every single AI doom argument that anyone ever makes.

Re: Hacked Nvidia 4090 GPU driver to enable P2P

#287

Post by int_19h »

Aerroon wrote: post #42
logicchains wrote: post #11

Earlier quoted context omitted.

Looks like we're only a few years away from a bona fide cyberpunk dystopia, in which only governments and megacorps are allowed to use AI, and hackers working on their own hardware face regular raids from the authorities.

I find it baffling that ideas like "govern compute" are even taken seriously. What the hell has happened to the ideals of freedom?! Does the government own us or something?

It's the same thing as always happens with freedom.

"But foreign propagandists ..."

"But extremists ..."

"But terrorists ..."

"But child abusers ..."

Re: Hacked Nvidia 4090 GPU driver to enable P2P

#288

Post by Sebb767 »

waldrews wrote: post #284

Would this approach be possible to extend downmarket, to older consumer cards? For a lot of LLM use cases we're constrained by memory and can tolerate lower compute speeds so long as there's no swapping. ELI5, what would prevent a hundred 1060-level cards from being used together?

> ELI5, what would prevent a hundred 1060-level cards from being used together?

In this case, you'd only have a single PCIe (v3!) lane per card, making the interconnect speed horribly slow. You'd also need to invest in thousands of dollars of hardware to get all of those cards connected and, unless power is free, you'd outspend any theoretical savings instantly.

In general, if you go back in card generations, you'll quickly hit so low memory limits amd slower compute that a modern CPU-based setup is better value.

Re: Hacked Nvidia 4090 GPU driver to enable P2P

#289

Post by int_19h »

Jerrrry wrote: post #78
erikbye wrote: post #68

Earlier quoted context omitted.

We already have export restrictions on cryptography. Of course there will be AI regulations.

>Of course there will be AI regulations. Are. As I and others have predicted, the executive order was passed defining a hard limit on the processing/compute power allowed without first 'checkin in' with the Letter boys. https://www.whitehouse.gov/briefing-room/presidential-action...

I wonder if we can squeeze a 20b parameter model into a book somehow...

Re: Hacked Nvidia 4090 GPU driver to enable P2P

#290

Post by samtheprogram »

pixelpoet wrote: post #266
samtheprogram wrote: post #239

Earlier quoted context omitted.

To me that was obvious satire of the crypto scene.

Ah yes, nothing like a bit of hypocrisy to make a point. It's okay though, as long as it's people we don't agree with, defrauding them is fine.

The website literally stated it was not for speculation, they didn't want the price to go up, and there were multiple ways to get some for free.

If people were reckless, greedy, and/or lazy because of the crypto hype and got "defrauded" without doing any amount of due diligence -- that's kinda the point.

Post reply on HN