Show HN: NN-512 – Generate standalone C code for neural nets
1–10 of 30 posts
Re: Show HN: NN-512 – Generate standalone C code for neural nets
#2NN-512 is an open-source Go program that generates fully AVX-512 vectorized, human-readable, stand-alone C implementations of convolutional neural nets
The generated C code is an example of AVX-512 programming using GCC's AVX-512 intrinsics. AVX-512 is exciting because its use of masking simplifies edge cases (partial loads, partial stores, etc.), there are 32 wide vector registers, and really excellent shuffle/permutation instructions (in particular, the two-input permute by var). Recent versions of GCC produce very good object code from C intrinsics
The goal of NN-512 is efficient neural net inference on inexpensive, CPU-only cloud instances. For example, a Skylake-X cloud compute instance costs $10 per CPU-core per month at Vultr, and the NN-512 generated code does about 18 DenseNet121 inferences per CPU-core per second (in series, not batched)
As AVX-512 becomes better supported by Intel and AMD chips, it becomes more attractive as an alternative to expensive GPU instances for workloads with small amounts of inference mixed with other computation
Re: Show HN: NN-512 – Generate standalone C code for neural nets
#3NN-512 is an open-source Go program that generates fully AVX-512 vectorized, human-readable, stand-alone C implementations of convolutional neural nets The generated C code is an example of AVX-512 programming using GCC's AVX-512 intrinsics. AVX-512 is exciting because its use of masking simplifies edge cases (partial loads, partial stores, etc.), there are 32 wide vector registers, and really excellent shuffle/permu…
Great idea.
Re: Show HN: NN-512 – Generate standalone C code for neural nets
#4NN-512 is an open-source Go program that generates fully AVX-512 vectorized, human-readable, stand-alone C implementations of convolutional neural nets The generated C code is an example of AVX-512 programming using GCC's AVX-512 intrinsics. AVX-512 is exciting because its use of masking simplifies edge cases (partial loads, partial stores, etc.), there are 32 wide vector registers, and really excellent shuffle/permu…
Out of curiosity, have you benchmarked this against some of the more standard NN libraries running on CPU?
Re: Show HN: NN-512 – Generate standalone C code for neural nets
#5NN-512 is an open-source Go program that generates fully AVX-512 vectorized, human-readable, stand-alone C implementations of convolutional neural nets The generated C code is an example of AVX-512 programming using GCC's AVX-512 intrinsics. AVX-512 is exciting because its use of masking simplifies edge cases (partial loads, partial stores, etc.), there are 32 wide vector registers, and really excellent shuffle/permu…
Out of curiosity, have you benchmarked this against some of the more standard NN libraries running on CPU?
[deleted]
Re: Show HN: NN-512 – Generate standalone C code for neural nets
#6NN-512 is an open-source Go program that generates fully AVX-512 vectorized, human-readable, stand-alone C implementations of convolutional neural nets The generated C code is an example of AVX-512 programming using GCC's AVX-512 intrinsics. AVX-512 is exciting because its use of masking simplifies edge cases (partial loads, partial stores, etc.), there are 32 wide vector registers, and really excellent shuffle/permu…
Is there the option to use AVX2 (256-bit) instead of AVX-512 (which can cause thermal throttling on basically every chip[a])? Now, if Intel can get AVX-512 working well, that’s something else.
Side question: do any Ryzen processors support AVX-512? AFAIK, they only support up to AVX2.
[a]: IIRC, some tests show AVX2 code actually being faster than the equivalent AVX-512 code because AVX2 doesn’t cause the processor to throttle so hard
Re: Show HN: NN-512 – Generate standalone C code for neural nets
#7Very nice. If you've tried using TFLite-Micro, you've encountered what C++ looks like in the hands of academics.
Any plans to support RNN layers?
Re: Show HN: NN-512 – Generate standalone C code for neural nets
#8NN-512 is an open-source Go program that generates fully AVX-512 vectorized, human-readable, stand-alone C implementations of convolutional neural nets The generated C code is an example of AVX-512 programming using GCC's AVX-512 intrinsics. AVX-512 is exciting because its use of masking simplifies edge cases (partial loads, partial stores, etc.), there are 32 wide vector registers, and really excellent shuffle/permu…
Is there the option to use AVX2 (256-bit) instead of AVX-512 (which can cause thermal throttling on basically every chip[a])? Now, if Intel can get AVX-512 working well, that’s something else. Side question: do any Ryzen processors support AVX-512? AFAIK, they only support up to AVX2. [a]: IIRC, some tests show AVX2 code actually being faster than the equivalent AVX-512 code because AVX2 doesn’t cause the processor t…
AVX2 also causes various degrees of thermal throttling depending on which chip you're using, such as Haswell. AVX-512 on Ice Lake (client) however is quite good and causes very little throttling versus its sustained speeds (e.g. my Ice Lake laptop @ 3.6GHz or whatever it is). Ultimately you have to do benchmarking yourself and if you care about inferences/sec you'll have to think about this stuff.
That said, AVX2 support would also be cool simply because it's more readily available to use on more platforms, not just my laptop...
Re: Show HN: NN-512 – Generate standalone C code for neural nets
#9I've heard that heavy sustained AVX3 / AVX-512 workloads have the potential to damage CPU's [0][1]. Would running/testing/playing around this software potentially risk damaging my CPU? Or is this only a risk when overclocking?
I'm not being facetious with this question at all - more genuine curiosity / responsible preventative caution for my own machines, because I'm very much interested in playing around with this if it's reasonable to do so.
Re: Show HN: NN-512 – Generate standalone C code for neural nets
#10NN-512 is an open-source Go program that generates fully AVX-512 vectorized, human-readable, stand-alone C implementations of convolutional neural nets The generated C code is an example of AVX-512 programming using GCC's AVX-512 intrinsics. AVX-512 is exciting because its use of masking simplifies edge cases (partial loads, partial stores, etc.), there are 32 wide vector registers, and really excellent shuffle/permu…
Out of curiosity, have you benchmarked this against some of the more standard NN libraries running on CPU?
No, the convolution generators were written to saturate the hardware
For example, under ideal conditions (no out-of-register memory access, no waiting on dependencies) you can sustain 27 single-precision FMADDs per CPU-core per cycle on a particular Skylake-X (i.e., approx 1.7 _mm512_fmadd_ps per cycle, each yielding 16 multiply-adds)
As soon as you start accessing memory, that number drops to about 20 FMADDs. With direct convolution methods (1x1 and arbitrary), the best you can do is achieve that, and NN-512 comes close
With the Fourier and Winograd convolutions, you start being limited by memory bandwidth, but the reduction in FMADDs that these methods provide means you end up ahead: your "effective" FMADD rate is much higher than what is possible through direct convolution. For example, NN-512 can exceed 48 effective FMADDs per cycle (on the 27 peak FMADD machine) with Winograd-Cook-Toom-Lavin, if the tensor is deep enough (enough channels)
So, NN-512 succeeds in saturating the hardware. Essentially all the time is spent in the matrix multiplications, doing FMADDs, or being blocked bringing half-precision weights into register for Fourier or Winograd
Until I generate a table of comparisons, you can use the previously stated number to do rough comparisons against the literature or other software packages: 18 DenseNet121 inferences per CPU-core per second on a cheap Skylake-X cloud instance