Live data from Hacker News

Hacked Nvidia 4090 GPU driver to enable P2P

github.com

361–365 of 365 posts

Re: Hacked Nvidia 4090 GPU driver to enable P2P

#361

Earlier quoted context omitted.

Demonstrably false: https://twitter.com/ESYudkowsky/status/1772624785672954115

Do they also take this position about biological intelligence? Because humans most certainly have an alignment problem too.

The concerning AGI properties include recursive self-improvement to superhuman capability levels and the ability to mass-deploy copies. Those are not on the horizon when it comes to humans. If hypothetically some human acquired such properties that would be equally concerning.

Re: Hacked Nvidia 4090 GPU driver to enable P2P

#363

Earlier quoted context omitted.

> The driver is not bypasses. This is a patch to the official open-source kernel-driver where the feature is added, which is how all upstream Linux driver development is done. [source code modification] is a weird way to describe software engineering. Making the code available for further development is kind of the whole point of open source. My previous comment was written with an unspoken assumption: Hardware drive…

> For ordinary free-and-open-source software, the source code availability largely guarantees community control. However, the same often does not apply to drivers. Even with source code availability, they're often written by vendors using NDAed information and in-house expertise about the underlying hardware design. As a result, drivers remain under a vendor's tight control. This is not true for any upstream Linux ke…

> This is not true for any upstream Linux kernel driver.

This is very true for at least a portion of upstream Linux kernel drivers.

> They are fully open source

True.

> fully controlled by the community

False (Of course, this claim is based on my belief that source code access is insufficient to do driver development, hardware datasheets are not just nice to have but mandatory for a true free operating system. If you disagree, nothing more can be said).

The upstream Linux kernel has absolutely no requirement that the contributed hardware drivers must include public documentation. In fact, many are written by vendor-hired contractors, or by independent embedded system companies who make computer gadgets based on these chips (so they have a business relationship with the vendor that give them datasheet access).

If you want to do any low-level work on these kinds of drivers independently, it almost always involves trial-and-error and educated guesses.

> and not subject to any NDAs to work on. The vendor can only exert control in the form of reviews and open maintainership. This is the case for both AMD and Intel GPU drivers.

Sure, the drivers themselves are not subject to any NDAs to work on, but hardware vendors often maintain a de-facto control - because they are in possession of technical advantages not available to outsiders, in terms of NDAed information and in-house expertise. Although theoretically everyone is free to participate, and it often happens, but when 80% of the experts are programmers with privileged information, it's difficult although not impossible for an outsider to join the development effort (it's no surprise that most outsider contributions are high-level changes that do not directly affect hardware operations, like moving away from a deprecated kernel API, fixing memory leaks, rather than, say, changing the PLL divider of the hardware's reference clock).

> For numbers, Arch Linux bundles some ~7.5k loadable kernel drivers (which is a subset of all available, but AUR only has 8 out-of-tree drivers, out of which two are proprietary and 2-3 are community-led. "vendor-controlled" drivers are an extreme outlier on Linux.

As a contributor to the Linux kernel, my personal experience is, drivers that contains code similar to the following are not rare:

    uint8_t registers[] = {
        0xFF, 0xFE, 0xEA, 0x3C, 0x5A, 0x6A,
        0x01, 0x02, 0x3A, 0x4D, 0x55, 0x66
        /* many lines */
    };

    /* initialize hardware */
    write_registers(device_ctx, registers);
I bet at least 10% of device drivers (or as high as 25%?) are written with privileged information and they're almost impossible for independent developers to work with (other than making educated guesses) due to incomplete or NDAed documentation. Datasheet access is the only way to interpret the intentions of this kind of drivers. Even with macro or bitfields definitions - which is better than none, the exact effect of a register is often subtle. Not to mention that hardware bugs are common and their workarounds often involve very particular register-write sequences - and that tend not to be publicly documented. Sometimes, even firmware binary blobs are embedded into the source code this way, and the code is almost always contributed by a vendor who actually knows what the blob is doing. If I was asked to do some development on top of an existing driver but without hardware documentation, I would refuse it every single time.

Re: Hacked Nvidia 4090 GPU driver to enable P2P

#364

Earlier quoted context omitted.

Is this the same guy that hacked the PS3?

He has a very checkered history with "hacking" things. He tends to build heavily on the work of others, then use it to shamelessly self-promote, often to the massive detriment of the original authors. His PS3 work was based almost completely on a presentation given by fail0verflow at CCC. His subsequent self-promotion grandstanding world tour led to Sony suing both him and fail0verflow, an outcome they were specifica…

[dead]

Re: Hacked Nvidia 4090 GPU driver to enable P2P

#365

Earlier quoted context omitted.

That is very interesting if tinygrad can support it! Every other library I've seen had the limitation on dividing the heads, so I'd (perhaps incorrectly) assumed that it's a general problem for inference.

There are some interesting hacks you can do like replicating the K/V weights by some factor which allows them to be evenly divisible by whatever number of gpus you have. Obviously there is a memory cost there, but it does work.

how could you go about testing this on say llama3 70b with two 4090s - vllm supports tensor parallelism, would the expectation be that inference would be faster with p2p? How would you update the nvidia driver tho? Thanks any thought appreciated
Post reply on HN