PyTorch 1.8, with AMD ROCm support • Hacker News Board

Re: PyTorch 1.8, with AMD ROCm support

#131

Post by jefft255 » Fri, Mar 05, 2021, 9:56 PM UTC

microtonal wrote: post #127

jefft255 wrote: post #88

Earlier quoted context omitted.

Not so sure about that. Here’s two things you can do (assuming you’re not training huge transformers or something). 1. Test your code with super low batch size. Bad for convergence, good for sanity check before submitting your job to a super computer. 2. Post-training evaluation. I’m pretty sure the M1 has enough power to do inference for not-so-big models. These two reasons are why I’m sometimes running stuff on my…

1. Test your code with super low batch size. Bad for convergence, good for sanity check before submitting your job to a super computer. Or you can buy a desktop machine for the same price as an M1 MacBook with 32GB or 64GB RAM and an RTX2060 or RTX3060 (which support mixed-precision training) and you can actually finetune a reasonable transformer model with a reasonable batch size. E.g., I can finetune a multi-task X…

Absolutely agree! My points were more about the benefits of running code on your own machine rather than in the cloud or on a cluster. I don’t own an M1, but if I did I wouldn’t want to use it to train models locally... When on my laptop I still deploy to my lab desktop; this adds little friction compared to a compute cluster, and as you mention we’re able to do interesting stuff with a regular gaming GPU. When everything works great and I now want to experiment at scale, I then deploy my working code to a supercomputer.

Re: PyTorch 1.8, with AMD ROCm support

#132

Post by ckastner » Fri, Mar 05, 2021, 11:12 PM UTC

slavik81 wrote: post #130

ckastner wrote: post #54

Earlier quoted context omitted.

Hardware support is key, though. CUDA works with basically any card made in the last 5 years, consumer or compute. ROCm seems to work with a limited set of compute cards only. There's a ROCm team in Debian [1] trying to push ROCm forward (ROCm being open), but just getting supported hardware alone is already a major roadblock, which stalls the effort, and hence any contributions Debian could give back. [1] https://sa…

I didn't know about the Debian team. Thank you very much for informing me! I'm not sure how much I can do, but I would be happy to discuss with them the obstacles they are facing and see what I can help with. I'm better positioned to help with software issues than hardware matters, but I would love to hear about ROCm from their perspective. What's the best way to contact the team?

The team's communication channel is the Debian AI mailing list [1], and they'd absolutely appreciate someone reaching out. Having ROCm succeed is a mutual interest.

From what I recall, on the software side, I think the difficulties stemmed from ROCm being distributed (naturally, of course) as a vendor would do it, for example: targeting specific kernels, compilers, distribution versions. The problem with this is that all these may become out of date, which means that the GitHub instructions may no longer work for anyone wanting to give it a try. This was the case for some of the team members attempting to rebuild some of the packages.

The Debian team would prefer to package all elements of the ROCm ecosystem and include them in the official Debian archive, so that users can just run `apt-get install rocm-something` without worrying about the kernel version etc., just as they can already do a trivial `apt-get install nvidia-cuda-toolkit`.

I think collaboration with AMD here would be mutually beneficial, as on the one hand, Debian wants FOSS accelerated computing, which AMD is pushing, and on the other hand, Debian is naturally experienced in distributing things for Debian, Ubuntu, etc., something AMD could benefit from.

Thank you for engaging!

[1] https://lists.debian.org/debian-ai/

Re: PyTorch 1.8, with AMD ROCm support

#133

Post by jampekka » Sat, Mar 06, 2021, 12:13 AM UTC

_coveredInBees wrote: post #104

jampekka wrote: post #69

Earlier quoted context omitted.

OTOH PyTorch seems to be highly explosive if you try to use it outside the mainstream use (i.e. neural networks). There's sadly no performant autodiff system for general purpose Python. Numba is fine for performance, but does not support autodiff. JAX aims to be sort of general purpose, but in practice it is quite explosive when doing something other than neural networks. A lot of this is probably due to supporting C…

I really don't understand the GPUs are overrated comment. As someone who uses Pytorch a lot and GPU compute almost every day, there is an order of magnitude difference in the speeds involved for most common CUDA / Open-CL accelerated computations. Pytorch makes it pretty easy to get large GPU accelerated speed-ups with a lot of code we used to traditionally limit to Numpy. And this is for things that have nothing to…

For a lot of cases you don't really need that much performance. Modern processors are plenty fast. It seems that current push to use GPU also pushes people towards GPU oriented solutions, such as using huge NNs for more or less anything, while other approaches would in many cases be magnitudes more efficient and robust.

GPUs (or "wide SIMDs" more generally) have quite profound limitations. Branching is very limited, recursion is more or less impossible and parallelism is possible only for identical operations. This makes for example many recursion-based time-series methods (e.g. Bayesian filtering) very tricky or practically impossible. From what I gather, running recurrent networks is also tricky and/or hacky on GPU.

GPUs are great for some quite specific, yet quite generally applicable, solutions, like tensor operations etc. But being tied to GPUs' inherent limitations also limits the space of approaches that are feasible to use. And in the long run this can stunt the development of different approaches.

Re: PyTorch 1.8, with AMD ROCm support

#134

Post by jampekka » Sat, Mar 06, 2021, 12:17 AM UTC

wxnx wrote: post #100

jampekka wrote: post #69

Earlier quoted context omitted.

OTOH PyTorch seems to be highly explosive if you try to use it outside the mainstream use (i.e. neural networks). There's sadly no performant autodiff system for general purpose Python. Numba is fine for performance, but does not support autodiff. JAX aims to be sort of general purpose, but in practice it is quite explosive when doing something other than neural networks. A lot of this is probably due to supporting C…

> I have nothing against supporting GPUs (although I think their use is overrated and most people would do fine with CPUs), but Python really needs a general purpose, high performance autodiff. As someone who works with machine learning models day-to-day (yes, some deep NNs, but also other stuff) - GPUs really seem unbeatable to me for anything gradient-optimization-of-matrices (i.e. like 80% of what I do) related. E…

For gradient-optimization-of-matrices for sure. Just make sure that you don't use gradient-optimization-of-matrices just because they run well on GPUs. There may well be more efficient approaches to your problems that are infeasible for the GPUs' wide SIMD architecture you may miss if you tie yourself to GPUs.

In general it's more that some specific models are easy for GPUs. Most models probably are not.

Re: PyTorch 1.8, with AMD ROCm support

#135

Post by _coveredInBees » Sat, Mar 06, 2021, 6:12 AM UTC

jampekka wrote: post #133

_coveredInBees wrote: post #104

Earlier quoted context omitted.

I really don't understand the GPUs are overrated comment. As someone who uses Pytorch a lot and GPU compute almost every day, there is an order of magnitude difference in the speeds involved for most common CUDA / Open-CL accelerated computations. Pytorch makes it pretty easy to get large GPU accelerated speed-ups with a lot of code we used to traditionally limit to Numpy. And this is for things that have nothing to…

For a lot of cases you don't really need that much performance. Modern processors are plenty fast. It seems that current push to use GPU also pushes people towards GPU oriented solutions, such as using huge NNs for more or less anything, while other approaches would in many cases be magnitudes more efficient and robust. GPUs (or "wide SIMDs" more generally) have quite profound limitations. Branching is very limited,…

I still don't get the criticism of Pytorch. If anything, you can get the best of both worlds in many way with their API supporting on GPU and on CPU operations in exactly the same ways.

Re: PyTorch 1.8, with AMD ROCm support

#136

Post by h0l0gr4ph1c » Sat, Mar 06, 2021, 6:29 AM UTC

slavik81 wrote: post #128

microtonal wrote: post #108

Earlier quoted context omitted.

At my previous employer, we bought two Radeon VIIs (in addition to NVIDIA GPUs). The last time I tried it (just over ~6 months ago), there were still many bugs. Things would just crash and burn very frequently (odd shape errors, random crashes, etc.). Two colleagues reported some of those bugs in ROCm, but the bug reports were largely ignored. Maybe out-of-the-box support for PyTorch will result in more polish. Who k…

So, just up front: these are my personal opinions. I do not speak on behalf of AMD as a company. I'm just a software developer who works on ROCm. I joined AMD specifically because I wanted to help ROCm succeed. If the problems you encountered are related to a particular ROCm software library, I would encourage you to open an issue on the library's GitHub page. You will get the best results if you can get your problem…

As someone who bought a RX580 for playing with Deep learning with ROCm (It was supported at the time). After posting to one or two bug threads, I had the same experience as the gp -- our issues were ignored. The issues have recently closed as the RX580 is no longer supported.

As for long term success, good luck, but once bitten twice shy.

Re: PyTorch 1.8, with AMD ROCm support

#137

Post by dheera » Sat, Mar 06, 2021, 6:55 PM UTC

codethief wrote: post #120

dheera wrote: post #41

Earlier quoted context omitted.

I'll also add a caveat that toolage for Jetson boards is extremely incomplete. They supply you with a bunch of sorely outdated models for TensorRT like Inceptionv3 and SSD-MobileNetv2 and VGG-16. WTF, it's 2021. If you want to use anything remotely state-of-the-art like EfficientDet or HRNet or Deeplab or whatever you're left in the dark. Yes you can run TensorFlow or PyTorch (thankfully they give you wheels for thos…

> I'll also add a caveat that toolage for Jetson boards is extremely incomplete. A hundred times this. I was about to write another rant here but I already did that[0] a while ago, so I'll save my breath this time. :) Another fun fact regarding toolage: Today I discovered that many USB cameras work poorly on Jetsons (at least when using OpenCV), probably due to different drivers and/or the fact that OpenCV doesn't su…

They provide some SSD-Mobilenet-v2 here:

https://github.com/dusty-nv/jetson-inference

Yeah, it works. I get 140 fps on a Xavier NX. It's super impressive for the wattage and size of the device. But they want you to train it using their horrid "DIGITS" interface, and it doesn't support any more recent networks.

I really wish Nvidia would stop trying to reinvent the wheel in training and focus on keeping up with being able to properly parse all the operations in the latest state-of-the-art networks which are almost always in Pytorch or TF 2.x.

Re: PyTorch 1.8, with AMD ROCm support

#138

Post by dheera » Sat, Mar 06, 2021, 7:01 PM UTC

codethief wrote: post #117

my123 wrote: post #46

Earlier quoted context omitted.

You have https://github.com/NVIDIA-AI-IOT/torch2trt as an option for example to use your own models on TensorRT just fine. And https://github.com/tensorflow/tensorrt for TF-TRT integration.

TF-TRT doesn't work nearly as well as pure TRT. On my Jetson Nano a 300x300 SSD-MobileNetV2 with 2 object classes runs at 5 FPS using TF, <10 FPS using TF-TRT and 30 FPS using TensorRT.

This. Try any recent network with TF-TRT and you'll find that memory is constantly being copied back and forth between TF and TRT components of the system every time it stumbles upon an operation not supported in TRT.

As such I often got slower results with TF-TRT than just pure TF, and at most a marginal improvement, even though what TRT does is conceptually awesome from a deployment standpoint, and if it only supported all the operations in TF, it could be a several-fold speed up in many cases.

Re: PyTorch 1.8, with AMD ROCm support

#139

Post by mpfundstein » Sat, Mar 06, 2021, 8:09 PM UTC

std_badalloc wrote: post #44

PyTorch is the most impressive piece of software engineering that I know of. So yeah, it's a nice interface for writing fast numerical code. And for zero effort you can change between running on CPUs, GPUs and TPUs. There's some compiler functionality in there for kernel fusing and more. Oh, and you can autodiff everything. There's just an incredible amount of complexity being hidden behind behind a very simple inter…

and you didn't even talk about data and model parallelism. which often just works out of the box

Re: PyTorch 1.8, with AMD ROCm support

#140

Post by mpfundstein » Sat, Mar 06, 2021, 8:10 PM UTC

jampekka wrote: post #133

_coveredInBees wrote: post #104

Earlier quoted context omitted.

I really don't understand the GPUs are overrated comment. As someone who uses Pytorch a lot and GPU compute almost every day, there is an order of magnitude difference in the speeds involved for most common CUDA / Open-CL accelerated computations. Pytorch makes it pretty easy to get large GPU accelerated speed-ups with a lot of code we used to traditionally limit to Numpy. And this is for things that have nothing to…

For a lot of cases you don't really need that much performance. Modern processors are plenty fast. It seems that current push to use GPU also pushes people towards GPU oriented solutions, such as using huge NNs for more or less anything, while other approaches would in many cases be magnitudes more efficient and robust. GPUs (or "wide SIMDs" more generally) have quite profound limitations. Branching is very limited,…

> For a lot of cases you don't really need that much performance. Modern processors are plenty fast. It seems that current push to use GPU also pushes people towards GPU oriented solutions, such as using huge NNs for more or less anything, while other approaches would in many cases be magnitudes more efficient and robust.

for instance?