From reading the fragment shader code, it seems to
a) sample a 4x4 grid of 2x2 neighborhoods
b) quantize the color to 3 bits per channel (so 512 colors)
c) "hash" (not really hashing) the "key" computed from adding up all the pixel neighborhoods and index into a lookup table: http://tllabs.io/asciistreetview/fontHash.png
d) use the green channel from the hash table to index into the font: http://tllabs.io/asciistreetview/8x8.png
It could be a little smarter about using the shape of the neighborhood to figure out which character to use, but it seems the lookup table is solely based on brightness, or I'm missing some subtlety in the code. It seems overly complex for what it does, but not for what it could be doing.