Build a Neural Network

Hm. I just finished teaching an ML course where all of the assignments were pure Python (on purpose, so students would actually have the chance to see all of the code). One of the assignments included implementing reverse-mode autodiff and a NN classifier on top. It can be done in ~600 lines of clear python, serious!

> I'd hope that those conflicts only occur on writes and not reads?

They happen on reads and writes.

The best way to describe it is... GPUs internally not only have tons of cores and threads (albeit "SIMD" threads, not true threads...)... they also have multiple memory banks.

I know AMD's architecture better. So let me talk about GCN. The smallest "cohesive" structure of an AMD GCN GPU is the execution unit. An execution unit has its own L1 data-cache, a 64kB "shared-memory" region, and finally the 256 vALUs (vector ALUs). There are 4-instruction pointers (64-simd threads per instruction pointer).

I'll focus on the high performance "64kB Shared Memory" region.

The 64kB shared memory is organized into 32-channels. In effect, channel 0 handles all addresses ending in XXXX00. Channel 1 handles all addresses ending in XXXX04. Channel 2 handles all addresses ending in XXXX08. Etc. etc. Channel 31 handles all addresses ending in XXXX7C.

Each channel can serve ONE thread per clock cycle. So if all 256-threads of the execution unit try to grab address 400000 (ending in 00, so channel 0), they all hammer channel 0. Channels 1 through 31 don't do anything, because no one asked for data from them. In this case, ONLY one channel is working, while 31-channels are sitting around doing nothing.

In this case, the last thread will be waiting for ~256 clock cycles, because its got 255 threads in front of it trying to grab data.

If you instead wrote your program so that all the threads asked for data "equally" between all 32-channels, then your code would be 32x faster (because all 32-channels would be working). Each channel has 8 requests (32x8 == 256 reads), and the 256 threads all get their data in just 8 clock cycles.

--------

This isn't a problem in CPU world, because your L1 cache on Intel / AMD systems only has one channel per core. Actually, AMD offers TWO L1 channels per core (!!), and Intel offers 3x channels per core (2x reads + 1x write). So your one thread can do 2-reads + 1x write per clock tick.

If you have a massive 32-core CPU, you still have 2x reads + 1x write per core. So total of 64-reads + 32 writes across the whole system. This makes CPUs simple.

GPUs however, have a compromise. The 256-simd threads (or really, up to 2560-threads on an AMD system per EU) share the same 32 read/write channels.

--------

Technically speaking, there are "channels" and "banks", two different memory organization schemes in a GPU. In practice, you can just pretend that only channels exist, because a bank conflict more or less acts the same as a channel conflict.

I second this suggestion; I took that course when it was called "ML Class" during the Fall of 2011 (yep, I was one of the guinea pigs for what became one of the first courses of Coursera). It was an excellent course.

Here's an example of what one student of the ML Class built, after being inspired by what he was learning and videos that played during the course:

https://blog.davidsingleton.org/nnrccar/

It kinda shocked me at the time, because I knew quite a bit about ALVINN from books and articles I had read as a teenager in the 80s and 90s. This guy had created the same thing using a cell phone and a cheap RC vehicle! Ok, there was also an Arduino and computer involved - but it really hit home the fact that technology around neural networks had advanced quite a bit!

I also took the other course, "AI Class", but due to personal issues I had to drop out about halfway through.

The next year, after Udacity started, they introduced a course similar to AI Class called "How to Build Your Own Self-Driving Vehicle" (it's called something else today - something like "Robotics and Artificial Intelligence 302" or something like that).

That class was done in Python, and taught me even more about AI/ML - with a focus towards self-driving vehicles of course. Things I learned about that I struggled with or had no real concepts of before:

1. SLAM (Simultaneous Localization and Mapping) 2. Path Finding algorithms (A* and the like) 3. Kalman Filtering (what it is for, how it works) 4. PID Algorithm (how to implement and tune it) 5. More neural network stuff...

...and many other things. Another very excellent and free course to take if you're interested in learning this stuff.

Broadcasts can be done efficiently on AMD systems. I dunno about NVidia, but I would assume NVidia PTX has some kind of low-level broadcast mechanism too.

A lot of optimization is just knowing all of the special ways you can move memory around. Broadcast was common enough that they've given AMD GPUs a special instruction just for it.

So in the case of neural networks all reading from the same input, you'd want to do it through the broadcast instructions, instead of through shared memory. Shared memory would create bank conflicts.