What I wish someone had told me about tensor computation libraries

In what sense is this "better"?

The generated code is like

    __m512i wfs16 = _mm512_castsi256_si512(_mm512_cvtps_ph(wf25, _MM_FROUND_TO_NEAREST_INT|_MM_FROUND_NO_EXC));
    fs16 = _mm512_inserti64x4(wfs16, _mm512_cvtps_ph(wf26, _MM_FROUND_TO_NEAREST_INT|_MM_FROUND_NO_EXC), 1);
    _mm512_mask_storeu_epi32(wfPtr1+230400+38400*i5+768*c2+128*k1+64*m2+16*f3, 3855, wfs16);
    _mm512_mask_storeu_epi32(wfPtr1+345584+38400*i5+768*c2+128*k1+64*m2+16*f3, 61680, wfs16);

(which is a set of 4 lines that appear in the middle of an ~800 line function).

That's not "human readable".

Sure you can use asan or gdb, but if gdb profiles slowly, what can you do? You're still at the mercy of the code generator to be able to optimize things.

Let me chip in with some self-promotion.

This book explains and executes every single line of code interactively, from low level operations to high-level networks that do everything automatically. The code is built on the state of the art performance operations of oneDNN (Intel, CPU) and cuDNN (CUDA, GPU). Very concise readable and understandable by humans.

https://aiprobook.com/deep-learning-for-programmers/

Here's the open source library built throughout the book:

https://github.com/uncomplicate/deep-diamond

Some chapters from the beginning of the book are available on my blog, as a tutorial series:

https://dragan.rocks

Google those _mm512_... intrinsics (they are part of GCC) to see what they mean. The code you pasted is converting single-precision floats to half-precision floats, and storing the half-precision floats to memory, 32 at a time. That's filter packing, which happens during initialization (and never during inference)

I agree, if you don't know anything about how convolution is implemented (filter packing, data packing, matrix multiplication, sum unpacking), you could be lost. But it's very shallow compared to a JIT or CUDA library scheme, and a knowledgeable ML performance engineer would have no difficulty

The inference function (at the end of the C file) is a series of blocks, each block corresponding to a convolution or other complex operation. It's straightforward to see which, by looking at where the weights come from (a field in a struct that has the same name as the layer in your graph)

If you use perf top (for example) you can see which convolution was most expensive, and why. Does the shape of the tensor produce many small partial blocks around the edge, so the packing is inefficient (a lot of tile overhang), for example? You can see that by glancing at the code and seeing that there are many optimized blocks around the edges. As a rule, if NN-512 generates small code for a tensor (few edge cases) you have chosen an efficient tensor shape, with respect to the tile

Or you might find that batch normalization is being done at inference time (as in DenseNet), instead of being integrated into the convolution weights (as in ResNet), because there's fanout from the source and a ReLU in between. You can see that easily in the generated code (the batch norm fmadd instructions will appear in the packing or unpacking code)

Is the matrix multiplication slow because there are too few channels per group (as in ResNeXt)? Easy to see in perf, make your groups bigger. Are you using an inefficient filter shape, so we have to fall back to a slower general purpose convolution? You can easily see whether Winograd or Fourier was used

And so on

I’m truly baffled as to why such a sophisticated and useful package is being distributed and advertised by an anonymous individual.

Here's my take about TF (in general, not particularly 1.x or 2.x):

Like many things from Google, I always had the impression that the library, while better than alternatives at the time, is too tailored to Google use cases. And if you fall outside of them, bad luck.

Still, at work we find it easier to deploy and interoperate with other tools than Pytorch. Hell, we have a guy working in Pytorch who converts his work to ONNX so that we can then connect those to some tooling we already have from back when TF was our only backend.

Could there be a better way? Perhaps. But we have to ship models and TF "just* works" (with a big asterisk, yeah).