Ryzen Threadripper Pro 3995WX Spotted

Thread divergence.

CUDA-cores are linked and take if-statements and for-loops together. This means that if one CUDA core takes a for-loop 1-million times, 31-other CUDA cores will take the 1-million loop with them. (32-cores per NVidia SM). EDIT: CUDA keeps things semantically correct per thread by "throwing away work" (execution mask is disabled, so those 31-other threads effectively execute NOPs), but the cores are wasted.

Worst case scenario, your CUDA cores accomplish 1/32th the work they could do, as they spin idle waiting for the last 1 thread to complete a for loop or its unique combination of if-statements.

Matrix multiplication doesn't have any thread divergence, because all threads loop the same amount of times on all workitems. This means that Tensors / Deep Learning simply ignores the problem, because they're written as matrix multiplication problems.

Avoiding thread-divergence, or mitigating its effects, is possible, but requires advanced programming skill that few have.

--------

Examples:

Chess -- Traditional multithreaded chess algorithms have every thread check out a different branch of the chess search tree. But because each branch has different positions and moves, its very difficult to code it in such a way that all threads are doing useful work.

Web Servers -- If every thread is handling a different request, then every thread probably traverses a different set of if-statements and loops. This is a thread-divergence nightmare.

Databases -- SQL Databases traditionally have each thread running a different, independent SQL query. Like web-servers, its difficult to imagine parallelism from a high-level.

-------

However, elements of Chess, Web-servers, and Databases can probably be efficiently parallelized onto GPUs.

Chess -- This github repo demonstrates that enumerating positions can be done GPU-parallel: https://github.com/ankan-ban/perft_gpu

Web-servers -- Text parsing, Regex, and many other problems common to Web-servers have been parallelized on GPUs. The main question for me, is whether or not the PCIe traversal would be worthwhile (lower bandwidth than RAM, high latency). Web-servers tend to be I/O heavy and less compute heavy... probably not dense enough to benefit from GPU compute.

Databases -- Merge-join and Hash-join probably can be parallelized to GPUs.

Figuring out the optimal CPU + GPU team is going to be a research problem over the next 10, maybe 20 years.

Its quite possible that someone figures out an x86 emulator on GPUs that fuzzes register values, similar to AVX512 fuzzing. https://gamozolabs.github.io/fuzzing/2018/10/14/vectorized_e...

------

Fuzzing is unique in that 99% of code probably traverses the same set of if-statements or branches. Whenever code significantly diverges (usually defined by Fuzzers as a significantly different instruction-pointer path of some kind), you've successfully fuzzed a new result.

Fuzzers... strangely enough... are programs I'd expect to run pretty well on GPUs or SIMD. If enough R&D research effort were put into it.