Learning to Optimize Tensor Programs
arxiv.org
Learning to Optimize Tensor Programs
1–10 of 14 posts
Re: Learning to Optimize Tensor Programs
#2Summary:
Tensor program is able to be optimized by using machine learning and transfer learning. The numerical program optimization model is trained on feature from low-level AST of the program.
Experiments:
Tasks: ResNet, MobileNet, LSTM LM, DQN
Hardware: CUDA/ARM GPU/ARM CPU
Speed up compare to CUDNN, TensorFlow Lite and ARMComputeLib: ~from 1.2X to 3.8X faster in end-to-end test.
Re: Learning to Optimize Tensor Programs
#3I'd have never thought that the art of low-level optimization would be the first one with some serious automation from Deep Learning... Very interesting paper!
Re: Learning to Optimize Tensor Programs
#4Summary: Tensor program is able to be optimized by using machine learning and transfer learning. The numerical program optimization model is trained on feature from low-level AST of the program. Experiments: Tasks: ResNet, MobileNet, LSTM LM, DQN Hardware: CUDA/ARM GPU/ARM CPU Speed up compare to CUDNN, TensorFlow Lite and ARMComputeLib: ~from 1.2X to 3.8X faster in end-to-end test.
Be careful what they benchmark against. The Maxwell/Pascal GEMM routines are _very_ optimized, since Nervana Systems (in their infancy) made an assembler for them which they showed off/demonstrated by implementing GEMM (single-precision dense matrix*matrix multiplication). It was lacking like 2% from the theoretical performance, as predicted by the clockrate, but they had no idea why these 2% were missing. They hand-scheduled the instructions to get around the limited number of register banks, and the associated bank conflicts, a concept normal CPUs afaik only see with DRAM. At that time, the official GEMM, albeit hand-optimized, only reached like 80% of the theoretical value.
Re: Learning to Optimize Tensor Programs
#5Compare to polyhedral optimization [0], supported in GCC via Graphite [1] and in LLVM via Polly [2].
These have a lower ceiling than hand-optimized assembler, but they automate the tuning of how to nest the loops for maximum benefits due to the cache hierarchy. Considering their ability to do so for general purpose number crunching loops, they are rather nice, but there is still some integration work needed, especially for LLVM, as lack a nice place in the existing optimization pipeline.
[0]: https://en.wikipedia.org/wiki/Polytope_model [1]: https://gcc.gnu.org/wiki/Graphite [2]: https://gcc.gnu.org/wiki/Graphite
Re: Learning to Optimize Tensor Programs
#6Summary: Tensor program is able to be optimized by using machine learning and transfer learning. The numerical program optimization model is trained on feature from low-level AST of the program. Experiments: Tasks: ResNet, MobileNet, LSTM LM, DQN Hardware: CUDA/ARM GPU/ARM CPU Speed up compare to CUDNN, TensorFlow Lite and ARMComputeLib: ~from 1.2X to 3.8X faster in end-to-end test.
Be careful what they benchmark against. The Maxwell/Pascal GEMM routines are _very_ optimized, since Nervana Systems (in their infancy) made an assembler for them which they showed off/demonstrated by implementing GEMM (single-precision dense matrix*matrix multiplication). It was lacking like 2% from the theoretical performance, as predicted by the clockrate, but they had no idea why these 2% were missing. They hand-…
the benchmarks are not about GEMM, but real-world deep learning workloads which could have very different characteristics from GEMM
Re: Learning to Optimize Tensor Programs
#7Earlier quoted context omitted.
Be careful what they benchmark against. The Maxwell/Pascal GEMM routines are _very_ optimized, since Nervana Systems (in their infancy) made an assembler for them which they showed off/demonstrated by implementing GEMM (single-precision dense matrix*matrix multiplication). It was lacking like 2% from the theoretical performance, as predicted by the clockrate, but they had no idea why these 2% were missing. They hand-…
the benchmarks are not about GEMM, but real-world deep learning workloads which could have very different characteristics from GEMM
I just wanted to caution that one has to be careful what one is comparing against, as the libraries got significant speed improvements over time, without that being widely advertised. So it matters a lot if one compares this library against CuDNN from 1 month ago, or to CuDNN from 2 years ago. The latter is _much_ slower.
The GEMM example was just there as the details of the optimization have been published, unlike most other hand-tuned assembler routines for DNN workloads.
Re: Learning to Optimize Tensor Programs
#8Earlier quoted context omitted.
the benchmarks are not about GEMM, but real-world deep learning workloads which could have very different characteristics from GEMM
I just wanted to caution that one has to be careful what one is comparing against, as the libraries got significant speed improvements over time, without that being widely advertised. So it matters a lot if one compares this library against CuDNN from 1 month ago, or to CuDNN from 2 years ago. The latter is _much_ slower. The GEMM example was just there as the details of the optimization have been published, unlike m…
CuDNN v7 was used in the experiments, in the experiments parts each comparison was listed with version or commit number.
Re: Learning to Optimize Tensor Programs
#9Compare to polyhedral optimization [0], supported in GCC via Graphite [1] and in LLVM via Polly [2]. These have a lower ceiling than hand-optimized assembler, but they automate the tuning of how to nest the loops for maximum benefits due to the cache hierarchy. Considering their ability to do so for general purpose number crunching loops, they are rather nice, but there is still some integration work needed, especial…
Polyhedral optimisation is cool (Facebook has been using it to great effect for ML kernels recently [1]), but it’s not the end of the story. It’s complementary to this paper, which seems to be about learning an effective and transferrable cost model to guide the optimisation process (you could use that learned cost model in a polyhedral optimiser).
Re: Learning to Optimize Tensor Programs
#10Compare to polyhedral optimization [0], supported in GCC via Graphite [1] and in LLVM via Polly [2]. These have a lower ceiling than hand-optimized assembler, but they automate the tuning of how to nest the loops for maximum benefits due to the cache hierarchy. Considering their ability to do so for general purpose number crunching loops, they are rather nice, but there is still some integration work needed, especial…
Polyhedral optimisation is cool (Facebook has been using it to great effect for ML kernels recently [1]), but it’s not the end of the story. It’s complementary to this paper, which seems to be about learning an effective and transferrable cost model to guide the optimisation process (you could use that learned cost model in a polyhedral optimiser). [1]: https://arxiv.org/abs/1802.04730
Uhh.... this sounds interesting. Maybe we'll see it soon-ish (<5y) in LLVM/GCC.