" Even then, while access to GPGPUs is now pretty much universal, GPGPU programming still feels like it is in the stone ages. Despite being a thing for over 10 years."
I'd be interested in reading an article from someone who has been doing it for 10 years as to why that is the case. I have theories but nowhere near enough direct experience to evaluate.
(My hypothesis is that the extreme parallelism makes it so very tiny mistakes have catastrophic performance impact by introducing accidental serialism, and as a result, it is very difficult to create an "easy to use" framework that doesn't abstract too much away and make it trivially easy to introduce even a tiny such error and crash performance. We actually make this mistake all the time in conventional CPU code, it just generally just costs you small integer multiples of performance instead of large integer multiples of performance.)