Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Use pytorch2+cu118 with ADA hardware for 50%+ speedup

gpux.ai

Post reply on HN
Print view View original

31–38 of 38 posts

Re: Use pytorch2+cu118 with ADA hardware for 50%+ speedup

#31

Post by vans554 »

brucethemoose2 wrote: post #7
vans554 wrote: post #2

I accidentally stumbled upon this and did not expect such a speedup. It seems anything less than cu118 does not properly support the RTX4090 (or H100). Bumping to cuda12.2 with pytorch2.0.1+cu118 made my SDXL go 50% faster and ESRGAN 80% faster on the 4090.

You can also run PyTorch cu121 nightly builds, These also allow `torch.compile` to function properly with dynamic input, which should net another 30%+ boost to SD.

Pretty interesting. Using nightly + cu121 im getting 8.18 it/s, another 5% improvement vs 7.78 that cu118 gave.

Re: Use pytorch2+cu118 with ADA hardware for 50%+ speedup

#32

Post by alfalfasprout »

Oh man, I deal with CUDA version nuances all the time. ML dependency management in particular is always extra fun. Between all the different CUDA, CuDNN, NCCL versions and versions of TF frameworks and numpy dependencies, etc. it can quickly become a mess.

We've started really investing into a better solution-- always interesting to see just how big a difference getting the right CUDA version for a given build of eg; torch is.

Re: Use pytorch2+cu118 with ADA hardware for 50%+ speedup

#33

Post by breakds »

Or if you are using Nix use https://github.com/nixvital/ml-pkgs

  outputs = { self, nixpkgs, ... }@inputs: {
    overlays = {
      dev = nixpkgs.lib.composeManyExtensions [
        inputs.ml-pkgs.overlays.torch-family
        # Add some other overlays
      ];
    };
  };
in your flake.nix and you can use pytorch 2.0.1 compiled with CUDA 11.8 running on your 4090s. Downside is that the first time you will have to compile it and it can be quite long.

Re: Use pytorch2+cu118 with ADA hardware for 50%+ speedup

#34

Post by VHRanger »

hospitalJail wrote: post #24
vans554 wrote: post #2

I accidentally stumbled upon this and did not expect such a speedup. It seems anything less than cu118 does not properly support the RTX4090 (or H100). Bumping to cuda12.2 with pytorch2.0.1+cu118 made my SDXL go 50% faster and ESRGAN 80% faster on the 4090.

This was one of the reasons I skipped the 4090. So few people have the technology that I knew I'd be spending significant time figuring out solutions to problems. The other reason is that I'd wait a few years and get some 6090 with 4x the VRAM.

I doubt future generations will up the consumerarket cards past 24gb.

They know it's a bottleneck for LM training and inference, so they'll want to extract value by reserving it for the professional line cards

Re: Use pytorch2+cu118 with ADA hardware for 50%+ speedup

#36

Post by magixx »

alfalfasprout wrote: post #32

Oh man, I deal with CUDA version nuances all the time. ML dependency management in particular is always extra fun. Between all the different CUDA, CuDNN, NCCL versions and versions of TF frameworks and numpy dependencies, etc. it can quickly become a mess. We've started really investing into a better solution-- always interesting to see just how big a difference getting the right CUDA version for a given build of eg;…

Anytime I want try out some ML stuff I run into this driver and app version hell. Have you given Docker a try? I'm considering going this route for portability reasons but don't know if it will actually help.

Re: Use pytorch2+cu118 with ADA hardware for 50%+ speedup

#37

Post by fnands »

magixx wrote: post #36
alfalfasprout wrote: post #32

Oh man, I deal with CUDA version nuances all the time. ML dependency management in particular is always extra fun. Between all the different CUDA, CuDNN, NCCL versions and versions of TF frameworks and numpy dependencies, etc. it can quickly become a mess. We've started really investing into a better solution-- always interesting to see just how big a difference getting the right CUDA version for a given build of eg;…

Anytime I want try out some ML stuff I run into this driver and app version hell. Have you given Docker a try? I'm considering going this route for portability reasons but don't know if it will actually help.

Docker is good, but I also haven't had any issues with conda. At least for most ML projects.

Re: Use pytorch2+cu118 with ADA hardware for 50%+ speedup

#38

Post by alfalfasprout »

magixx wrote: post #36
alfalfasprout wrote: post #32

Oh man, I deal with CUDA version nuances all the time. ML dependency management in particular is always extra fun. Between all the different CUDA, CuDNN, NCCL versions and versions of TF frameworks and numpy dependencies, etc. it can quickly become a mess. We've started really investing into a better solution-- always interesting to see just how big a difference getting the right CUDA version for a given build of eg;…

Anytime I want try out some ML stuff I run into this driver and app version hell. Have you given Docker a try? I'm considering going this route for portability reasons but don't know if it will actually help.

Docker comes with its own problems (especially if you need to mutate the image under the hood). And multi arch quickly becomes a pain. Also much harder to make it truly portable.

We're rolling something around conda+conda-pack and plan to contribute it upstream. Look out for a blog post on HN later this year :)

Post reply on HN