Learning to Optimize Tensor Programs • Hacker News Board

Re: Learning to Optimize Tensor Programs

#11

Post by namibj » Tue, May 22, 2018, 10:27 PM UTC

antinucleon wrote: post #8

namibj wrote: post #7

Earlier quoted context omitted.

I just wanted to caution that one has to be careful what one is comparing against, as the libraries got significant speed improvements over time, without that being widely advertised. So it matters a lot if one compares this library against CuDNN from 1 month ago, or to CuDNN from 2 years ago. The latter is _much_ slower. The GEMM example was just there as the details of the optimization have been published, unlike m…

CuDNN v7 was used in the experiments, in the experiments parts each comparison was listed with version or commit number.

Well, I didn't RTFA. This was not meant to be specific for this article though.

Re: Learning to Optimize Tensor Programs

#12

Post by evrydayhustling » Wed, May 23, 2018, 2:07 AM UTC

Great example of recursive self improvement.

Re: Learning to Optimize Tensor Programs

#13

Post by empiricus » Wed, May 23, 2018, 7:08 AM UTC

Does this mean that it is easy now to get optimized deep learning kernels for AMD GPUs?

Re: Learning to Optimize Tensor Programs

#14

Post by jules » Wed, May 23, 2018, 9:11 AM UTC

namibj wrote: post #10

sidereal wrote: post #9

Earlier quoted context omitted.

Polyhedral optimisation is cool (Facebook has been using it to great effect for ML kernels recently [1]), but it’s not the end of the story. It’s complementary to this paper, which seems to be about learning an effective and transferrable cost model to guide the optimisation process (you could use that learned cost model in a polyhedral optimiser). [1]: https://arxiv.org/abs/1802.04730

Uhh.... this sounds interesting. Maybe we'll see it soon-ish (<5y) in LLVM/GCC.

It already exists! Check out Polly for LLVM.