Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Hacked Nvidia 4090 GPU driver to enable P2P

github.com

Post reply on HN
Print view View original

331–340 of 365 posts

Re: Hacked Nvidia 4090 GPU driver to enable P2P

#331

Post by segfaultbuserr »

_zoltan_ wrote: post #242
segfaultbuserr wrote: post #69

Earlier quoted context omitted.

> but there's nothing "suboptimal non-default options" about it If "bypassing the official driver to invoke the underlying hardware feature directly through source code modification (and incompatibilities must be carefully worked around by turning off IOMMU and large BAR, since the feature was never officially supported)" does not count as "suboptimal non-default options", then I don't know what counts as "suboptimal…

I have some news for you: you must disable IOMMU on the H100 platform anyway, at least for optimal GDS :-)

I stand corrected. If it's already suboptimal in practice to begin with, the hack does not more it more suboptimal... Still, disabling large BAR size is still sub-optimal...

Re: Hacked Nvidia 4090 GPU driver to enable P2P

#332

Post by segfaultbuserr »

schlauerfox wrote: post #280
jprete wrote: post #98

Earlier quoted context omitted.

If AI is actually capable of fulfilling all the capabilities suggested by people who believe in the singularity, it has far more capacity for harm than nuclear weapons. I think most people who are strongly pro-AI/pro-acceleration - or, at any rate, not anti-AI - believe that either (A) there is no control problem (B) it will be solved (C) AI won't become independent and agentic (i.e. it won't face evolutionary pressu…

This is all just Pascal's wager anyway.

Surely, there are many realistic ways to abuse this powerful technology, the creation of a self-aware Paperclip Maximizer is not necessarily required.

Re: Hacked Nvidia 4090 GPU driver to enable P2P

#333

Post by boromi »

andersa wrote: post #80

Incredible! I'd been wondering if this was possible. Now the only thing standing in the way of my 4x4090 rig for local LLMs is finding time to build it. With tensor parallelism, this will be both massively cheaper and faster for inference than a H100 SXM. I still don't understand why they went with 6 GPUs for the tinybox. Many things will only function well with 4 or 8 GPUs. It seems like the worst of both worlds now…

Any chance you could share the details of the build you'd go for. I need a server for our lab, but am kinda out of my depth with all the options/

Re: Hacked Nvidia 4090 GPU driver to enable P2P

#334

Post by faeriechangling »

corn13read2 wrote: post #94
andersa wrote: post #80

Incredible! I'd been wondering if this was possible. Now the only thing standing in the way of my 4x4090 rig for local LLMs is finding time to build it. With tensor parallelism, this will be both massively cheaper and faster for inference than a H100 SXM. I still don't understand why they went with 6 GPUs for the tinybox. Many things will only function well with 4 or 8 GPUs. It seems like the worst of both worlds now…

A macbook is cheaper though

Buying a MacBook for AI is great if you were already going to buy a MacBook, as this makes it a lot more cost competitive. It's also great if what you're doing is REALLY privacy sensitive, such as if you're a lawyer, where uploading client data to OpenAI is probably not appropriate or legal.

But in general, I find the appeal is narrow because either consumer GPUs are better for training in general and inferencing at scale[1]. Cloud services also allow the vast majority of individuals to get higher quality inferencing at lower cost. The result is Apple Silicon's appeal being quite niche.

[1] Mind you, Nvidia considers this a licensing violation, not that GeoHot has historically ever been all scared to violate a EULA and force a company to prove its terms have legal force.

Re: Hacked Nvidia 4090 GPU driver to enable P2P

#335

Post by faeriechangling »

papichulo2023 wrote: post #216
numpad0 wrote: post #172

Earlier quoted context omitted.

4x32GB(128GB) DDR4 is ~$250. 4x48GB(192GB) DDR5 is ~$600. Those are even cheaper than upgrade options for Macs($1k).

No many consumer mobo support 192GB DDR5.

Most consumer mobo's I see support this even if the setup isn't on the QVL. If a DDR5 motherboard support 4 sticks at all you can probably run 192gb on it so long as you update the BIOS firmware. The problem is running at rated speeds.

AMD tends to be worse than Intel, and I hear people having to run anywhere between DDR5-3200 to DDR5-5200. You are better off running two sticks, because even with 2 sticks you really can't run larger models with acceptable performance anyways, much less with 4.

There is competition to apple on the low end (dual channel fast DDR5) and on the high end (8+ channel like Xeon/Epyc/AmpereOne). In the middle, Apple is sort of crushing because if you run a true 4 channel system you're going to get poor performance if you load up a 192gb model, and if you compare pricing to 96gb/128gb apple systems, there's not all that much of a cost advantage and you have to make a lot of sacrifices to get there. The truth is that Apple really doesn't have all that much competition right now and won't for the foreseeable future.

Re: Hacked Nvidia 4090 GPU driver to enable P2P

#336

Post by stale2002 »

FeepingCreature wrote: post #328
stale2002 wrote: post #324

Earlier quoted context omitted.

> If the only evidence for AI doom you will accept is actual AI doom No actually. This is another mistake that the AI doomers make. They pretend like a demand for evidence means that the world has to end first. Instead, what would be perfectly good evidence, would be evidence of significant incremental harm that requires regulation on its own, independent of any doom argument. In between "the world literally ends by…

Well, at least if you see escalating measurable harm you'll come around, I'm happy about that. You won't necessarily get the escalating harm even if AI doom is real though, so you should try to discover if it is real even in worlds where hard takeoff is a thing. > What I would expect is for the people who claim to care about AI doom to actually be trying to measure real world harm. Why bother? If escalating harm is a…

> You won't necessarily get the escalating harm even if AI doom is real though

Yes we would. Unless you are one of those people who think that the magic doom nanobots are going to be invented overnight.

My comparisions to someone who is worried about literal magic, from harry potter, is apt.

But at that point, if you are worried about magic showing up instantly, then your position is basically not falsifiable. You can always retreat to some untestable, unfalsifiable magic.

Like there is actually nothing I could say, no evidence I could show to ever convince someone out of that position.

On the other hand, my position is actually fasifiable. There is absolutely all sorts of non world ending evidence that could convince me to think that AI is dangerous.

But nobody on the doomer side seems to care about any of that. Instead they invent positions that seem almost tailor made to avoid being falsifiable or disprovable so that they can continue to believe them despite any evidence to the contrary.

As in, if I were to purposeful invent an idea or philosophy that is impossible to be disproved or convinced out of the "I can't show you evidence because the world will end" position is what I would invent.

> you'll come around,

Do you admit that you won't though? Do you admit that no matter what evidence is shown to you, that you can just retreat and say that the magic could happen at any time?

Or even if this isn't you literally, that someone in your position could dismiss all counter evidence, no matter what, and nobody could convince someone out of that with evidence?

I am not sure how someone could ever possibly engage with you seriously on any of this, if that is your position.

Re: Hacked Nvidia 4090 GPU driver to enable P2P

#337

Post by papichulo2023 »

faeriechangling wrote: post #335
papichulo2023 wrote: post #216

Earlier quoted context omitted.

No many consumer mobo support 192GB DDR5.

Most consumer mobo's I see support this even if the setup isn't on the QVL. If a DDR5 motherboard support 4 sticks at all you can probably run 192gb on it so long as you update the BIOS firmware. The problem is running at rated speeds. AMD tends to be worse than Intel, and I hear people having to run anywhere between DDR5-3200 to DDR5-5200. You are better off running two sticks, because even with 2 sticks you really…

Hopefully Qualcom will free us of this 2 channels noghtmare.

Re: Hacked Nvidia 4090 GPU driver to enable P2P

#338

Post by papichulo2023 »

ojbyrne wrote: post #236
papichulo2023 wrote: post #216

Earlier quoted context omitted.

No many consumer mobo support 192GB DDR5.

A lot that have specs showing they support a max of 4x32 DDR5 actually support 4x48 DDR5 via recent BIOS updates.

In the specs yeap, in practice hardly anyone got it working. As far as I saw in reddit, it requires customizing timings to make 4 slots work over 6000 Mhz at the same time.

Re: Hacked Nvidia 4090 GPU driver to enable P2P

#339

Post by the8472 »

stale2002 wrote: post #336
FeepingCreature wrote: post #328

Earlier quoted context omitted.

Well, at least if you see escalating measurable harm you'll come around, I'm happy about that. You won't necessarily get the escalating harm even if AI doom is real though, so you should try to discover if it is real even in worlds where hard takeoff is a thing. > What I would expect is for the people who claim to care about AI doom to actually be trying to measure real world harm. Why bother? If escalating harm is a…

> You won't necessarily get the escalating harm even if AI doom is real though Yes we would. Unless you are one of those people who think that the magic doom nanobots are going to be invented overnight. My comparisions to someone who is worried about literal magic, from harry potter, is apt. But at that point, if you are worried about magic showing up instantly, then your position is basically not falsifiable. You ca…

> Like there is actually nothing I could say, no evidence I could show to ever convince someone out of that position.

There is, it is just very hard to obtain. Various formal proofs would do. On upper bounds. On controllability. On scalability of safety techniques.

The manhattan project scientists did check whether they'd ignite the atmosphere before detonating their first prototype. Yes, that was much simpler task. But there's no rule in nature that says proving a system to be safe must be as easy as creating the system. Especially when the concern is that the system adaptive and adversarial.

Recursive self-improvement is a positive feedback loop, like nuclear chain reactions, like virus replication. So if we have an AI that can program then we better make sure that it either cannot sustain such a positive feedback loop or that it remains controllable beyond criticality. Given the complexity of the task it appears unlikely that a simple ten-page paper proving this will show up on arxiv. But if one did that'd be great.

>> You won't necessarily get the escalating harm even if AI doom is real though

> Yes we would.

So what does guarantee a visible catastrophe that won't be attributed to human operators using a non-agentic AI incorrectly? We keep scaling and the systems will be treated as assistants/optimizers and it's always the operators fault. Until we roughly reach human-level on some relevant metrics. And at that point there's a very narrow complexity range from idiot to genius (human brains don't vary by orders of magnitude!). So as far as hardware goes this could be a very narrow range and we could shoot straight from "non-agentic sub-human AI" to "agentic superintelligence" in short timescales once the hardware has that latent capacity. And up until that point it will always have been a human error, lax corporate policies, insufficient filtering of the training set or whatever.

And it's not that it must happen this way. Just that there doesn't seem anything ruling it and similar pathways out.

Re: Hacked Nvidia 4090 GPU driver to enable P2P

#340

Post by mhh__ »

thot_experiment wrote: post #273
mhh__ wrote: post #158

Earlier quoted context omitted.

nvidia's software is their moat

That's a huge overstatement, it's a big part of the moat for sure, but there are other significant components (hardware, ecosystem lock-in, heavy academic incentives)

No software -> hardware is massively hobbled. Evidence: AMD.

Ecosystem -> Software. At the moment especially people are looking for arbitrages everywhere i.e. inference costs / being able to inference at all (llama.cpp)

Academics -> Also software but easily fiddled with a bit of spending as you say.

Post reply on HN