How do I, with a very nice AMD GPU, and running on Windows, play around with any sort of GPU accelerated learning for anything? Last I knew, pretty much nothing worked with either of those caveats.
For me, the easiest thing has been to set up a google cloud compute server with a good gpu. They start you off with a $300 credit good for one year and charge like $0.50/hour to run. You can get pretty solid models running in the scale of minutes. Spend your time learning, not setting up.