Earlier quoted context omitted.
GPUs are mainly good at doing matrix multiplication and other math with little branching. Compilers have a lot of if/else statements; training neural networks does not. IIRC (and I may be wrong) the architecture is something like many GPU cores share things like the code cache and branch prediction, so if the different cores have to execute different code paths there is a big performance decrease. Edit: another reply…
Also GPUs really like predictable memory access patterns so that they can load the data that'll be needed next ahead of time into registers or dedicated caches explicitly. They don't have cache hierarchy features a CPU has to minimize the average disruption from only partially predictable memory accesses and they don't have out of order windows to paper over the occasional low level cache miss.
pattern like the one below will execute up to 10 times faster if array is sorted due to data locality and branch prediction
sum: float
a: float array[some_huge_number]
fill_array(a, random(1))
for f in a
if a[i]>0.5
sum=sum+f