AMD may get across the CUDA moat • Hacker News Board

Re: AMD may get across the CUDA moat

#211

Post by mrweasel » Sat, Oct 07, 2023, 6:30 AM UTC

nextaccountic wrote: post #191

javchz wrote: post #26

CUDA is the only reason I have an Nvidia card, but if more projects start migrating to a more agnostic environment, I'll be really grateful. Running Nvidia in Linux isn't as much fun. Fedora and Debian can be incredibly reliable systems, but when you add an Nvidia card, I feel like I am back in Windows Vista with kernel crashes from time to time.

> CUDA is the only reason I have an Nvidia card, but if more projects start migrating to a more agnostic environment, I'll be really grateful. What AMD really needs is to have 100% feature parity with CUDA without changing a single line of code. Maybe for this to happen it needs to add hardware features or something (I see people saying that CUDA as an API is very tailored to the capabilities of nvidia GPUs), I don't…

> I see people saying that CUDA as an API is very tailored to the capabilities of nvidia GPUs

I'm wondering how true that is, because that could give NVidia issues in the future if they need to redesign their GPU should they hit some limit with the current designs. Dependence on certain instruction makes sense, but there's not technical preventing AMD from implementing those instructions, only legal mumbo jumbo.

Re: AMD may get across the CUDA moat

#212

Post by latchkey » Sat, Oct 07, 2023, 6:30 AM UTC

incognition wrote: post #200

omneity wrote: post #52

I was able to use ROCm recently with Pytorch and after pulling some hair it worked quite well. The Radeon GPU I had on hand was a bit old and underpowered (RDNA2) and it only supported matmul on fp64, but for the job I needed done I saw a 200x increase in it/s over CPU despite the need to cast everywhere, and that made me super happy. Best of all is that I simply set the device to `torch.device('cuda')` rather than o…

Fp64??

https://en.wikipedia.org/wiki/Double-precision_floating-poin...

NVIDIA fp32 (H100) has 2x more TFLOPS than AMD's fp32 (MI250) and AI doesn't need fp64 precision.

Re: AMD may get across the CUDA moat

#213

Post by bornfreddy » Sat, Oct 07, 2023, 6:36 AM UTC

hot_gril wrote: post #80

fluxem wrote: post #65

I call it the 90% problem. If AMD works for 90% of my projects, I would still buy NVIDIA, which works for 100%, even though I’m paying a premium

I'm lazy, so it's 99% for me. I don't even mess with AMD CPUs; I know they're not exactly the same instruction set as Intel, and more importantly they work with a different (and less mainstream) set of mobos, so I don't want em. If AMD manages to pull more customers their way, that's great, it just means lower Intel premium for me.

That's an interesting take. AMD mobos are no "less mainstream" than Intel ones are... When you choose a CPU you are also choosing a compatible mobo chipset. The companies that make motherboards are mostly the same, so there should be no big difference between those.

Also, while the CPU instruction sets are not exactly equal, the same is true for Intel processors of different generations too. And it doesn't matter one bit... Unless there is a bug in CPU you will never notice the difference, because it is taken care of at the compiler / kernel level.

Intel does have some advantages (and disadvantages too) over AMD, just not those.

Re: AMD may get across the CUDA moat

#214

Post by josephg » Sat, Oct 07, 2023, 6:57 AM UTC

mgaunard wrote: post #193

mikepurvis wrote: post #121

Earlier quoted context omitted.

Oh yeah, I mean... having the source for the container build is kind of table stakes at this point. No one would accept a 10gb mystery meat blob as the basis of their production system. It's bad enough that we still accept binary-only drivers and proprietary libraries like TensorRT. I think my issue is more just with the mindset that it's okay to have one narrow slice of supported versions of everything that are "kno…

I don't know what world you live in, but this is a problem for any software development. You need to ensure that there is only one version of any library used globally throughout the code and that the set of versions is compatible with each other, and preferably you also want everything to be built against the same toolchain with the same flags. That usually means onboarding third-party libraries into your own build…

It’s not a universal problem. A lot of modern languages allow multiple versions of a library to be pulled in to the same code base, through different dependency paths. (Eg nodejs, rust). It’s not a perfect answer by any means, but it’s nice not needing to worry about some package pulling in an inconvenient version of one of its dependencies.

Also, just to name it, it’s ridiculous that a specific graphics card manages to restrict the version of gproto that you’re using. You don’t have this problem with nvidia drivers, since cuda stuff is much less fiddly. AMD needs to pull a finger out and fix the bugs in their stack that make it so fragile like this.

Re: AMD may get across the CUDA moat

#215

Post by Roark66 » Sat, Oct 07, 2023, 7:06 AM UTC

I think the article claiming "PyTorch has dropped the drawbridge on the CUDA moat" is way over optimistic. Jest pytorch is widely used by researchers and by users to quickly iterate various over various ways to use the models, but when it comes to inference there are huge gains to be had by going a different route. Llama.cpp has showed 10x speedups on my hardware for example (32gb of gpu ram + 32gb of cpu ram)for models like falcon-40b-instruct, for much smaller models on the cpu (under 10b) I saw up to 3x speedup just by switching to onnc and openvino.

Apple has showed us in practice the benefits of CPU/GPU memory sharing, will AMD be able to follow in their footsteps? The article claims AMD has a design with up to 192gb of shared ram. Apple is already shipping a design with the same amount of RAM(if you can afford it). I wish them-and) success, but I believe they need to aim higher than just matching apple in some unspecified future.

Re: AMD may get across the CUDA moat

#216

Post by benreesman » Sat, Oct 07, 2023, 7:15 AM UTC

I know a lot of people don’t like George, I dislike plenty of people who are doing the right thing thing (including by some measures sama and siebel while they were pushing YC forward).

But not admitting the tinygrad project is the best Rebel Alliance on this is just a matter of letting vibe overcome results.

Re: AMD may get across the CUDA moat

#217

Post by bornfreddy » Sat, Oct 07, 2023, 7:18 AM UTC

lhl wrote: post #114

Havoc wrote: post #57

Earlier quoted context omitted.

>if this unicorn were to show up A unicorn like that showed up a couple hours ago. Someone posted a guide for getting llama to run on a 7900xtx https://old.reddit.com/r/LocalLLaMA/comments/170tghx/guide_i... It's still slow and janky but this really isn't that far away. I don't buy that AMD can't make this happen if they actually tried. Go on fiverr, get them to compile a list of top 100 people in the DIY LLM space,…

llama.cpp, ExLlama, and MLC LLM have all had ROCm inferencing for months (here are a bunch of setup instructions I've written up, for Linux and Windows: https://llm-tracker.info/books/howto-guides/page/amd-gpus ) - but I don't think that's the problem (and wouldn't drive lots of volume or having downstream impact in any case). The bigger problem is on the training/research support. Eg, here's no official support for…

Wow, that is really awkward... AMD should be donating the cards and even paying extra for the privilege - this is an important step for getting satisfied consumers. I hope they notice and rectify this situation so that Debian (and with it all downstream distros, like Ubuntu) can provide better support for their cards. I mean, that's a no-brainer...

Re: AMD may get across the CUDA moat

#218

Post by physicsguy » Sat, Oct 07, 2023, 7:19 AM UTC

Don’t agree at all. PyTorch is one library - yes, it’s important that it supports AMD GPUs but it’s not enough.

The ROCm libraries just aren’t good enough currently. The documentation is poor. AMD need to heavily invest in their software ecosystem around it, because library authors need decent support to adopt it. If you need to be a Facebook sized organisation to write an AMD and CUDA compatible library then the barrier to entry is too high.

Re: AMD may get across the CUDA moat

#219

Post by anuraaga » Sat, Oct 07, 2023, 7:24 AM UTC

mgaunard wrote: post #193

mikepurvis wrote: post #121

Earlier quoted context omitted.

Oh yeah, I mean... having the source for the container build is kind of table stakes at this point. No one would accept a 10gb mystery meat blob as the basis of their production system. It's bad enough that we still accept binary-only drivers and proprietary libraries like TensorRT. I think my issue is more just with the mindset that it's okay to have one narrow slice of supported versions of everything that are "kno…

I don't know what world you live in, but this is a problem for any software development. You need to ensure that there is only one version of any library used globally throughout the code and that the set of versions is compatible with each other, and preferably you also want everything to be built against the same toolchain with the same flags. That usually means onboarding third-party libraries into your own build…

I'd say with semver becoming far better known, this is not a problem for "any" software development. The developer gets the choice to pick libraries that are stable, often also influencing language choice. Mistakes happen, Guava broke the Java ecosystem for about two years, but it's never something that is accepted as just a fact of software development, it is a mistake.

Wanting to hold Python+C ecosystem more accountable is fair I think, at least from my own experience around half a year ago, Anaconda doesn't work and you need a Dockerfile for any sort of reproducibility, which can have issues since GPU with docker isn't that easy. And this means developers from the vendors working with Anaconda, for example, on solving the issue rather than just hoping for contributors to do it. If AMD were to make easy, reproducible builds without root or VM a reality, that would be reason enough to try their hardware. If not, hopefully Nvidia does and then there really would be no way across the moat for me at least.

Re: AMD may get across the CUDA moat

#220

Post by blueboo » Sat, Oct 07, 2023, 7:25 AM UTC

upbeat_general wrote: post #198

This article doesn’t address the real challenge [in my mind]. Framework support is one thing, but what about the million standalone CUDA kernels that have been written, especially common in research. Nobody wants to spend time re-writing/porting those, especially when they probably don’t understand the low-level details in the first place. Not to mention, what is the plan for comprehensive framework support? I’ve exp…

Research kernels mostly turn to ash upon publication anyway. The wheel turns and the next post-doc gives ROCm a try and we move on