Earlier quoted context omitted.
Cuda is a proprietary Nvidia framework. You don't need cuda support to train deep learning models on TensorFlow. ROCm also works - and is built by AMD and is open-source.
Have you tried installing rocm? Last I checked two months ago you had to run a Ubuntu 16.04 container on Ubuntu 18.04 because the amd drivers were only compiled for bionic and the rocm software was only compiled for xenial.
https://github.com/nixos-rocm/nixos-rocm
The last time we purchased GPUs, we bought two Radeon VIIs besides RTX 5000s. For training transformers in Tensorflow and PyTorch, they were a steal, because the training is only marginally slower than the RTXes at a fraction of the price.
For LSTMs, we have found performance regressions compared to NVIDIA, but it's largely because it was hard to hit the optimized MIOpen code paths.