General hardware. I used to do all my work on dedicated hardware with support for Lisp (CADRs, 36xxs, D-machines, and before that PDP-10s) because those general purpose machines couldn't implement important features like generational garbage collection without specific hardware assist. Then one day in 84/85 I saw a generational GC using the MMU on a 68K and the light dawned. Thus while I'm pretty interested in ML chi…
I wouldn't bet on the ML hardware startups either, but don't you think that what we regard as general hardware is always going to be too heavyweight to be suitable for massive parallelisation?
Also look at the workload: often many of the primary cores will be idle when you're crunching a big dataset.
Those two factors suggest that you want to use CPU hardware for mult-add fops, especially when you consider using smaller float sizes (not just f16 but even f8). And then you consider that the compiler can perhaps properly interleave these ops with the regular instruction mix...
This is who general purpose von Neumann architectures ate hardware, both RISC and CISC.
Now the current crop is pretty dire (e.g. AVX* is hard to use with a typical workload) but perhaps these won't be typical workload? Or the power/timing problems will be solved other ways.