AMD may get across the CUDA moat • Hacker News Board

Re: AMD may get across the CUDA moat

#241

Post by mgaunard » Sat, Oct 07, 2023, 9:51 AM UTC

anuraaga wrote: post #219

mgaunard wrote: post #193

Earlier quoted context omitted.

I don't know what world you live in, but this is a problem for any software development. You need to ensure that there is only one version of any library used globally throughout the code and that the set of versions is compatible with each other, and preferably you also want everything to be built against the same toolchain with the same flags. That usually means onboarding third-party libraries into your own build…

I'd say with semver becoming far better known, this is not a problem for "any" software development. The developer gets the choice to pick libraries that are stable, often also influencing language choice. Mistakes happen, Guava broke the Java ecosystem for about two years, but it's never something that is accepted as just a fact of software development, it is a mistake. Wanting to hold Python+C ecosystem more accoun…

Semver is a joke and doesn't work. Languages like C and C++ can easily have problems if you link code built with different versions together (even if you aim for them to be compatible, or even if they are indeed the same source version but with subtly different flags), and there are no good solutions for this, except not doing it.

A docker container is not really any different from any other process; the main difference is that it runs in a chroot pretty much.

Re: AMD may get across the CUDA moat

#242

Post by JonChesterfield » Sat, Oct 07, 2023, 9:54 AM UTC

KeplerBoy wrote: post #108

wyldfire wrote: post #104

Earlier quoted context omitted.

> Best of all is that I simply set the device to `torch.device('cuda')` rather than openCL, which does wonders for compatibility Man oh man where did we go wrong that cuda is the more compatible option over OpenCL?

It must be a misnomer on PyTorch's side. Clearly it's neither CUDA nor OpenCL. AMD should just get it's shit together. This is ridiculous. Not the name, but the fact that you can only do FP64 on a GPU. Everybody is moving to FP16 and AMD is stuck on doubles?

FP64 is what HPC is built on. F32 works on the cards too (same rate or faster). I don't know the status of F16 or F8.

Some architectures provide fast F16->F32 and F32->F16 conversion instructions so you can DIY the memory bandwidth saving - that always seemed reasonable to me, but I don't know if the AMD hardware people are/will go down that path.

Re: AMD may get across the CUDA moat

#243

Post by fransje26 » Sat, Oct 07, 2023, 10:01 AM UTC

incognition wrote: post #200

omneity wrote: post #52

I was able to use ROCm recently with Pytorch and after pulling some hair it worked quite well. The Radeon GPU I had on hand was a bit old and underpowered (RDNA2) and it only supported matmul on fp64, but for the job I needed done I saw a 200x increase in it/s over CPU despite the need to cast everywhere, and that made me super happy. Best of all is that I simply set the device to `torch.device('cuda')` rather than o…

Fp64??

Hardware limitation.

Re: AMD may get across the CUDA moat

#244

Post by nologic01 » Sat, Oct 07, 2023, 10:06 AM UTC

If the AI hype persists the CUDA moat will be less relevant in ~2 yrs.

Historically HPC was simply not sufficiently interesting (in commercial sense) for people to throw serious resources in the direction of making it a mass market capability.

NVIDIA first capitalized on the niche crypto industry (which faded) and was then well positioned to jump into the AI hype. The question is how much of the hype will become real business.

The critical factor for the post-CUDA world is not any circumstantial moat but who will be making money servicing stable, long term computing needs. I.e., who will be buying this hardware not with speculative hot money but with cashflow from clients that regularly use and pay for a HPC-type application.

These actors will be the long term buyers of commercially relevant HPC and they will have quite a bit of influence on this market.

Re: AMD may get across the CUDA moat

#245

Post by JonChesterfield » Sat, Oct 07, 2023, 10:19 AM UTC

nextaccountic wrote: post #191

javchz wrote: post #26

CUDA is the only reason I have an Nvidia card, but if more projects start migrating to a more agnostic environment, I'll be really grateful. Running Nvidia in Linux isn't as much fun. Fedora and Debian can be incredibly reliable systems, but when you add an Nvidia card, I feel like I am back in Windows Vista with kernel crashes from time to time.

> CUDA is the only reason I have an Nvidia card, but if more projects start migrating to a more agnostic environment, I'll be really grateful. What AMD really needs is to have 100% feature parity with CUDA without changing a single line of code. Maybe for this to happen it needs to add hardware features or something (I see people saying that CUDA as an API is very tailored to the capabilities of nvidia GPUs), I don't…

The idea was supposed to be people convert cuda to hip, which is a pretty similar language, either by hand or by running a tool called 'hipify' that comes with rocm. You can then compile that unmodified for amdgpu or for nvptx.

I think where that idea goes wrong is in order to compile it unmodified for nvptx, you need to use a toolchain which knows hip and nvptx, which the cuda toolchain does not. Clang can mostly compile cuda successfully but it's far less polished than the cuda toolchain. ROCm probably has the nvptx backend disabled, and even if it's built in, best case it'll work as well as clang upstream does.

What I'm told does work is keeping all the source as cuda and using hipify as part of a build process when using amdgpu - something like `cat foo.cu | hipify | clang -x hip -` - though I can't personally vouch for that working.

The original idea was people would write in opencl instead of cuda but that really didn't work out.

Re: AMD may get across the CUDA moat

#246

Post by JonChesterfield » Sat, Oct 07, 2023, 10:38 AM UTC

javchz wrote: post #208

nextaccountic wrote: post #191

Earlier quoted context omitted.

> CUDA is the only reason I have an Nvidia card, but if more projects start migrating to a more agnostic environment, I'll be really grateful. What AMD really needs is to have 100% feature parity with CUDA without changing a single line of code. Maybe for this to happen it needs to add hardware features or something (I see people saying that CUDA as an API is very tailored to the capabilities of nvidia GPUs), I don't…

I think that could work too. I wonder if they could do a translation layer, something like Apple with the M1 chips that translates JIT x86 to ARM.

That's a fun idea. Qemu parses a binary into something very like a compiler IR, optimises it a bit, then writes it out as a binary for the same or another target in JIT like fashion. So that sort of thing can be built. Apple's rosetta is functionally similar, I expect it does the same sort of thing under the hood. Valgrind is another from the same architecture.

It would be a painful reverse engineering process - the cuda file format is sort of like elf, but with undocumented bonus constraints, and you'd have to reverse the instruction encoding to get sass, which isn't documented, or try to take it directly to ptx which is somewhat documented, and then convert that onward.

It would be far more difficult than compiling cuda source directly. I'm not sure anyone would pay for a cuda->amdgpu conversion tool, and it's hard to imagine AMD making one as part of ROCm.

Re: AMD may get across the CUDA moat

#247

Post by JonChesterfield » Sat, Oct 07, 2023, 10:45 AM UTC

throwitawayfam wrote: post #31

IntelMiner wrote: post #27

Earlier quoted context omitted.

Didn't he do what he always does. Rake in a ton of money, fart around and then cash out exclaiming it's everyone else's fault? The way he stole Fail0verflow's work with the PS3 security leak after failing to find a hypervisor exploit for months absolutely soured any respect I had for him at the time

Yep, did exactly that. IMO he threw a fit, even though AMD was working with him squashing bugs. https://github.com/RadeonOpenCompute/ROCm/issues/2198#issuec...

That's a tough issue to read through, thanks for the link. 'Your demo code on a system setup exactly as you describe dereferences null in the kernel and falls over'. Fuzz testing + a vaguely reasonable kernel debugging workflow should make things like that much harder to find.

Re: AMD may get across the CUDA moat

#248

Post by weebull » Sat, Oct 07, 2023, 11:06 AM UTC

mikepurvis wrote: post #121

xahrepap wrote: post #105

Earlier quoted context omitted.

I know it's still different than what you're looking for, so you probably already know this, but many projects like this have the Dockerfile on github which shows exactly how they set up the image. For example: https://github.com/RadeonOpenCompute/ROCm-docker/blob/master... They also have some for Fedora. Looks like for this you need to install their repo: curl -sL https://repo.radeon.com/rocm/rocm.gpg.key | apt-key…

Oh yeah, I mean... having the source for the container build is kind of table stakes at this point. No one would accept a 10gb mystery meat blob as the basis of their production system. It's bad enough that we still accept binary-only drivers and proprietary libraries like TensorRT. I think my issue is more just with the mindset that it's okay to have one narrow slice of supported versions of everything that are "kno…

> I think my issue is more just with the mindset that it's okay to have one narrow slice of supported versions of everything that are "known to work together" and those are what's in the container and anything outside of those and you're immediately pooched.

I hear you. I think docker has been a plague on the quality of software. It's allowed "works for me" to become the norm, except it's now pronounced "works on the official docker image". It seems to be especially true in the ML sphere where compiling things is so temperamental that there's a lot of binaries being distributed.

Docker was meant to be a deployment platform, not a distribution medium.

Re: AMD may get across the CUDA moat

#249

Post by KeplerBoy » Sat, Oct 07, 2023, 11:21 AM UTC

JonChesterfield wrote: post #242

KeplerBoy wrote: post #108

Earlier quoted context omitted.

It must be a misnomer on PyTorch's side. Clearly it's neither CUDA nor OpenCL. AMD should just get it's shit together. This is ridiculous. Not the name, but the fact that you can only do FP64 on a GPU. Everybody is moving to FP16 and AMD is stuck on doubles?

FP64 is what HPC is built on. F32 works on the cards too (same rate or faster). I don't know the status of F16 or F8. Some architectures provide fast F16->F32 and F32->F16 conversion instructions so you can DIY the memory bandwidth saving - that always seemed reasonable to me, but I don't know if the AMD hardware people are/will go down that path.

Sure but Radeon cards are not HPC accelerators. A modest 7800XT for example, which would be a great card for SD, has 76 TFlops@FP16, 37TF@FP32 and 1.16TF@FP64.

Keeping all those FPUs busy is another problem and not easy, but in cases where it can be done FP32 is clearly desirable.

Re: AMD may get across the CUDA moat

#250

Post by hankman86 » Sat, Oct 07, 2023, 11:24 AM UTC

I suspect that AMD will use their improved compatibility with the leading ML stack for data center deals. Presumably by offering steep discounts over NVIDIA’s GPUs. This might help them to break into the market.

Individual ML practitioners will probably not be tempted to switch to AMD cards anytime soon. Whatever the price difference is: it will hardly offset the time that is subsequently sunk into working around remaining issues resulting from a non-CUDA (and less mature) stack underneath PyTorch.