AMD funded a drop-in CUDA implementation built on ROCm: It's now open-source • Hacker News Board

Re: AMD funded a drop-in CUDA implementation built on ROCm: It's now open-source

#411

Post by andrecarini » Tue, Feb 13, 2024, 8:34 PM UTC

swozey wrote: post #197

bick_nyers wrote: post #162

Earlier quoted context omitted.

The latest version of CUDA is 12.3, and version 12.2 came out 6 months prior. How many people are running an older version of CUDA right now on NVIDIA hardware for whatever particular reason? Even if AMD lagged support on CUDA versioning, I think it would be widely accepted if the performance per dollar at certain price points was better. Taking the whole market from NVIDIA is not really an option, it's better to att…

I just went through this this weekend - If you're running in Windows and want to use deepspeed, you have to still use Cuda 12.1 because deepspeed 13.1 is the latest that works with 12.1. There's no deepspeed for windows that works with 12.3. I tried to get it working this weekend but it was a huge PITA so I switched to putting everything into WSL2 then in arch on there pytorch etc in containers so I could flip versio…

> I'm scared to mount the vhdx right now [...] I have EVERYTHING I need [...] sitting in a chezmoi git repo ON THE VM.

You probably already know but just in case you don't: you can set up a Linux VM with VirtualBox on your Windows and then mount the vhdx (read-only) as an additional disk to extract the stuff you need via shared folders.

Re: AMD funded a drop-in CUDA implementation built on ROCm: It's now open-source

#412

Post by Cu3PO42 » Tue, Feb 13, 2024, 9:17 PM UTC

sophrocyne wrote: post #354

Cu3PO42 wrote: post #335

Earlier quoted context omitted.

Hi! I really appreciate you taking the time to reply. I have since gotten Invoke to run and was already able to get some results I'm really quite happy with, so thank you for your time and commitment working on Invoke! I understand that ROCm is still challenging, but it seems my problems were less related to ROCm or Invoke itself and more to Python dependency management. It really boiled down to getting the correct (…

You bet - Thanks for the feedback. Glad you're enjoying Invoke! We do have Docker packages hosted on GH, but I'll be the first to admit that we haven't prioritized ROCm. Contributors who have AMDs are a scant few, but maybe we'll find some help in wrangling that problem now that we know there's an avenue to do so.

As promised in my other comment, I did send a PR! https://github.com/invoke-ai/InvokeAI/pull/5714

Re: AMD funded a drop-in CUDA implementation built on ROCm: It's now open-source

#413

Post by westurner » Tue, Feb 13, 2024, 10:31 PM UTC

HarHarVeryFunny wrote: post #406

westurner wrote: post #269

Earlier quoted context omitted.

> Proton+DXVK for Linux gaming "Building the DirectX shader compiler better than Microsoft?" (2024) https://news.ycombinator.com/item?id=39324800 E.g. llama.cpp already supports hipBLAS; is there an advantage to this ROCm CUDA-compatibility layer - ZLUDA on Radeon (and not yet Intel OneAPI) - instead or in addition? https://github.com/ggerganov/llama.cpp?tab=readme-ov-file#hi... https://news.ycombinator.com/item?id=3…

BLAS will only get you so far. About the highest level operation it has is matmul, which you can use to build convolution (im2col, matmul, col2im), but that won't be as performant as a hand optimized cuDNN convolution kernel. Same goes for any other high level neural net building blocks - trying to build them on top of BLAS will not get you remotely close to performance of a custom kernel. What's nice about BLAS is t…

"CUDNN API supported by HIP" has a coverage table: https://rocm.docs.amd.com/projects/HIPIFY/en/amd-staging/tab...

ROCm/hipDNN wraps CuDNN on Nvidia and MiOpen on AMD; but hasn't been updated in awhile: https://github.com/ROCm/hipDNN

https://news.ycombinator.com/item?id=37808036 : conda-forge has various BLAS implementations, including MKL-optimized BLAS, and compatible NumPy and SciPy builds.

BLAS: Basic Linear Algebra Sub programs: https://en.wikipedia.org/wiki/Basic_Linear_Algebra_Subprogra...

"Using CuPy on AMD GPU (experimental)" https://docs.cupy.dev/en/v13.0.0/install.html#using-cupy-on-... :

  $ sudo apt install hipblas hipsparse rocsparse rocrand rocthrust rocsolver rocfft hipcub rocprim rccl

Re: AMD funded a drop-in CUDA implementation built on ROCm: It's now open-source

#414

Post by HarHarVeryFunny » Wed, Feb 14, 2024, 12:20 AM UTC

westurner wrote: post #413

HarHarVeryFunny wrote: post #406

Earlier quoted context omitted.

BLAS will only get you so far. About the highest level operation it has is matmul, which you can use to build convolution (im2col, matmul, col2im), but that won't be as performant as a hand optimized cuDNN convolution kernel. Same goes for any other high level neural net building blocks - trying to build them on top of BLAS will not get you remotely close to performance of a custom kernel. What's nice about BLAS is t…

"CUDNN API supported by HIP" has a coverage table: https://rocm.docs.amd.com/projects/HIPIFY/en/amd-staging/tab... ROCm/hipDNN wraps CuDNN on Nvidia and MiOpen on AMD; but hasn't been updated in awhile: https://github.com/ROCm/hipDNN https://news.ycombinator.com/item?id=37808036 : conda-forge has various BLAS implementations, including MKL-optimized BLAS, and compatible NumPy and SciPy builds. BLAS: Basic Linear Alge…

I guess I misunderstood you.

You were asking if this CUDA compatability layer might hold any advantage over HIP (e.g. for use by llama.cpp) ?

I think the answer is no, since HIP includes pretty full-featured support for many of the higher level CUDA-based APIs (cuDNN, cuBLAS, etc), while per the Phoronix article ZLUDA only (currently) has minimal support for them.

I wouldn't expect ZLUDA to provide any performance benefit over HIP either, since on AMD hardware HIP is just a pass-thru to MIOpen (AMD's equivalent to cuDNN), rocBLAS, etc.

Re: AMD funded a drop-in CUDA implementation built on ROCm: It's now open-source

#415

Post by westurner » Wed, Feb 14, 2024, 12:53 AM UTC

Cu3PO42 wrote: post #335

sophrocyne wrote: post #307

Earlier quoted context omitted.

Hey there - I'm a maintainer (and CEO) of Invoke. It's something we're monitoring as well. ROCm has been challenging to work with - we're actively talking to AMD to keep apprised of ways we can mitigate some of the more troublesome experiences that users have with getting Invoke running on AMD (and hoping to expand official support to Windows AMD) The problem is that a lot of the solutions proposed involve significan…

Hi! I really appreciate you taking the time to reply. I have since gotten Invoke to run and was already able to get some results I'm really quite happy with, so thank you for your time and commitment working on Invoke! I understand that ROCm is still challenging, but it seems my problems were less related to ROCm or Invoke itself and more to Python dependency management. It really boiled down to getting the correct (…

> AMD's ROCm OCI base images,

ROCm docs > "Install ROCm Docker containers" > Base Image: https://rocm.docs.amd.com/projects/install-on-linux/en/lates... links to ROCm/ROCm-docker: https://github.com/ROCm/ROCm-docker which is the source of docker.io/rocm/rocm-terminal: https://hub.docker.com/r/rocm/rocm-terminal :

  docker run -it --device=/dev/kfd --device=/dev/dri --group-add video rocm/rocm-terminal

ROCm docs > "Docker image support matrix": https://rocm.docs.amd.com/projects/install-on-linux/en/lates...

ROCm/ROCm-docker//dev/Dockerfile-centos-7-complete: https://github.com/ROCm/ROCm-docker/blob/master/dev/Dockerfi...

Bazzite is a ublue (Universal Blue) fork of the Fedora Kinoite (KDE) or Fedora Silverblue (Gnome) rpm-ostree Linux distributions; ublue-os/bazzite//Containerfile : https://github.com/ublue-os/bazzite/blob/main/Containerfile#... has, in addition to fan and power controls, automatic updates on desktop, supergfxctl, system76-scheduler, and an fsync kernel:

  rpm-ostree install rocm-hip \
        rocm-opencl \
        rocm-clinfo

But it's not `rpm-ostree install --apply-live` because its a Containerfile.

To install a ublue-os distro, you install any of the Fedora ostree distros: {Silverblue, Kinoite, Sway Atomic, or Budgie Atomic} from e.g. a USB stick and then `rpm-ostree rebase `:

  rpm-ostree rebase ostree-unverified-registry:ghcr.io/ublue-os/bazzite:stable
  rpm-ostree rebase ostree-unverified-registry:ghcr.io/ublue-os/bazzite-nvidia:stable
  rpm-ostree rebase ostree-image-signed:

ublue-os/config//build/ublue-os-just/40-nvidia.just defines the `ujust configure-nvidia` and `ujust toggle-nvk` commands: https://github.com/ublue-os/config/blob/main/build/ublue-os-...

There's a default `distrobox` with pytorch in ublue-os/config//build/ublue-os-just/etc-distrobox/apps.ini: https://github.com/ublue-os/config/blob/main/build/ublue-os-...

  [mlbox]
  image=nvcr.io/nvidia/pytorch:23.08-py3
  additional_packages="nano git htop"
  init_hooks="pip3 install huggingface_hub tokenizers transformers accelerate datasets wandb peft bitsandbytes fastcore fastprogress watermark torchmetrics deepspeed"
  pre-init-hooks="/init_script.sh"
  nvidia=true
  pull=true
  root=false
  replace=false

docker.io/rocm/pytorch: https://hub.docker.com/r/rocm/pytorch

pytorch/builder//manywheel/Dockerfile: https://github.com/pytorch/builder/blob/main/manywheel/Docke...

ROCm/pytorch//Dockerfile: https://github.com/ROCm/pytorch/blob/main/Dockerfile

The ublue-os (and so also bazzite) OCI host image Containerfile has Sunshine installed; which is a 4k HDR 120fps remote desktop solution for gaming.

There's a `ujust remove-sunshine` command in system_files/desktop/shared/usr/share/ublue-os/just/80-bazzite.just : https://github.com/ublue-os/bazzite/blob/main/system_files/d... and also kernel args for AMD:

  pstate-force-enable:
    rpm-ostree kargs --append-if-missing=amd_pstate=active

ublue-os/config//Containerfile: https://github.com/ublue-os/config/blob/main/Containerfile

LizardByte/Sunshine: https://github.com/LizardByte/Sunshine

moonlight-stream https://github.com/moonlight-stream

Anyways, hopefully this PR fixes the immediate issue: https://github.com/invoke-ai/InvokeAI/pull/5714/files

conda-forge/pytorch-cpu-feedstock > "Add ROCm variant?": https://github.com/conda-forge/pytorch-cpu-feedstock/issues/...

And Fedora supports OCI containers as host images and also podman container images with just systemd to respawn one or a pod of containers.

Re: AMD funded a drop-in CUDA implementation built on ROCm: It's now open-source

#416

Post by Cu3PO42 » Wed, Feb 14, 2024, 8:43 AM UTC

westurner wrote: post #415

Cu3PO42 wrote: post #335

Earlier quoted context omitted.

Hi! I really appreciate you taking the time to reply. I have since gotten Invoke to run and was already able to get some results I'm really quite happy with, so thank you for your time and commitment working on Invoke! I understand that ROCm is still challenging, but it seems my problems were less related to ROCm or Invoke itself and more to Python dependency management. It really boiled down to getting the correct (…

> AMD's ROCm OCI base images, ROCm docs > "Install ROCm Docker containers" > Base Image: https://rocm.docs.amd.com/projects/install-on-linux/en/lates... links to ROCm/ROCm-docker: https://github.com/ROCm/ROCm-docker which is the source of docker.io/rocm/rocm-terminal: https://hub.docker.com/r/rocm/rocm-terminal : docker run -it --device=/dev/kfd --device=/dev/dri --group-add video rocm/rocm-terminal ROCm docs > "Dock…

I actually used the rocm/pytorch image you also linked.

I'm not sure what you're pointing to with your reference to the Fedora-based images. I'm quite happy with my NixOS install and really don't want to switch to anything else. And as long as I have the correct kernel module, my host OS really shouldn't matter to run any of the images.

And I'm sure it can be made to work with many base images, my point was just that the dependency management around pytorch was in a bad state, where it is extremely easy to break.

> Anyways, hopefully this PR fixes the immediate issue: https://github.com/invoke-ai/InvokeAI/pull/5714/files

It does! At least for me. It is my PR after all ;)

Re: AMD funded a drop-in CUDA implementation built on ROCm: It's now open-source

#417

Post by rasz » Wed, Feb 14, 2024, 10:12 AM UTC

>For reasons unknown to me, AMD decided this year to discontinue funding the effort and not release it as any software product.

classic AMD

Re: AMD funded a drop-in CUDA implementation built on ROCm: It's now open-source

#418

Post by westurner » Wed, Feb 14, 2024, 2:17 PM UTC

Cu3PO42 wrote: post #416

westurner wrote: post #415

Earlier quoted context omitted.

> AMD's ROCm OCI base images, ROCm docs > "Install ROCm Docker containers" > Base Image: https://rocm.docs.amd.com/projects/install-on-linux/en/lates... links to ROCm/ROCm-docker: https://github.com/ROCm/ROCm-docker which is the source of docker.io/rocm/rocm-terminal: https://hub.docker.com/r/rocm/rocm-terminal : docker run -it --device=/dev/kfd --device=/dev/dri --group-add video rocm/rocm-terminal ROCm docs > "Dock…

I actually used the rocm/pytorch image you also linked. I'm not sure what you're pointing to with your reference to the Fedora-based images. I'm quite happy with my NixOS install and really don't want to switch to anything else. And as long as I have the correct kernel module, my host OS really shouldn't matter to run any of the images. And I'm sure it can be made to work with many base images, my point was just that…

Unfortunately NixOS (and Debian and Ubuntu) lack SELinux policies or other LSM implementations by default out of the box, and container-selinux contains more than e.g. docker.

Is there a way to 'restorecon --like / /nix/os/root72`; to apply SELonix extended filesystem attributes labels just to NixOS prefixes?

Some research is done with RPM-based distros; which have become so advanced with rpm-ostree support.

FWICS Bazzite has NixOS support, too; in addition to distrobox containers.

Bazzite has alot of other stuff installed that's not necessary when attempting to isolate sources of variance in the interest of reproducible research; but being for gaming it has various optimizations.

InvokeAI might be faster to install and to compute with with conda-forge builds.

Re: AMD funded a drop-in CUDA implementation built on ROCm: It's now open-source

#419

Post by weebull » Wed, Feb 14, 2024, 9:02 PM UTC

croutons wrote: post #358

weebull wrote: post #299

Earlier quoted context omitted.

Define "viable"?

A backend that runs PyTorch out of the box and is as easy to setup / use as nvidia stack.

Installing PyTorch with the PyTorch website instructions for AMD was pretty painless for me on Linux. I know everybodies experience is different, but install wasn't the issue for me.

For me the issue on AMD was stability in situations when VRAM was getting tight.

Re: AMD funded a drop-in CUDA implementation built on ROCm: It's now open-source

#420

Post by weebull » Wed, Feb 14, 2024, 9:06 PM UTC

Keyframe wrote: post #319

weebull wrote: post #296

Earlier quoted context omitted.

> Personally I want Nvidia to break the x86-64 monopoly The one supplied by two companies?

Maybe he meant homogeneity which Nvidia did try and tries with Arm.. but, on the other hand how wild would it be for Nvidia to enter x86-64 as well? It's probably never going to happen due to licensing if nothing else, lest we remember nForce chipset ordeal with intel legal.

Indeed, but I think people forget that the reason AMD have a license in the first place was because Intel's customers in the early days required a second source for it's processors.

Who owns the Cyrix x86 license these days?