DLVM: A modern compiler framework for neural network DSLs
1–10 of 20 posts
Re: DLVM: A modern compiler framework for neural network DSLs
#2Why are all the neural network DSLs JIT obsessed?
Re: DLVM: A modern compiler framework for neural network DSLs
#3Why are all the neural network DSLs JIT obsessed?
Well, the success of neural nets over the past few years has come through harnessing massive processing power.
The problem is a lot of the programming can be low level and ad-hoc. I think the idea of the various DSLs is to allow the model to be compactly specified while having the programs go as fast as possible. A JIT may be one way to accomplish this.
Re: DLVM: A modern compiler framework for neural network DSLs
#4What does it mean by modern?
Re: DLVM: A modern compiler framework for neural network DSLs
#5Current tally of high-performance, deep-learning-oriented DSLs/IRs/compilers, in no particular order:
- TensorComprehensions (Facebook): https://github.com/facebookresearch/TensorComprehensions
- XLA (Google): https://www.tensorflow.org/performance/xla/
- taco (MIT): http://tensor-compiler.org/
- DLVM (UIUC): http://dlvm.org/
- nGraph (Intel): http://ngraph.nervanasys.com/docs/cpp/
- TVM (DMLC): https://github.com/dmlc/tvm
Honorable mention to Julia (http://julialang.org) as well.
Re: DLVM: A modern compiler framework for neural network DSLs
#6Why are all the neural network DSLs JIT obsessed?
Well, the success of neural nets over the past few years has come through harnessing massive processing power. The problem is a lot of the programming can be low level and ad-hoc. I think the idea of the various DSLs is to allow the model to be compactly specified while having the programs go as fast as possible. A JIT may be one way to accomplish this.
Relying upon a JIT often means the ability to create things which preclude the use of static compilers, which means that accelerated hardware, like ours, cannot be used efficiently.
Re: DLVM: A modern compiler framework for neural network DSLs
#7Current tally of high-performance, deep-learning-oriented DSLs/IRs/compilers, in no particular order: - TensorComprehensions (Facebook): https://github.com/facebookresearch/TensorComprehensions - XLA (Google): https://www.tensorflow.org/performance/xla/ - taco (MIT): http://tensor-compiler.org/ - DLVM (UIUC): http://dlvm.org/ - nGraph (Intel): http://ngraph.nervanasys.com/docs/cpp/ - TVM (DMLC): https://github.com/dm…
As far as I know Tile/PlaidML (Vertex.AI) is the only DSL+compiler that's usable for real workloads across a variety of hardware. https://github.com/plaidml/plaidml
Re: DLVM: A modern compiler framework for neural network DSLs
#8Why are all the neural network DSLs JIT obsessed?
Lots of modern models have very late binding variables
which are hard to precompile for (sentence length in MNT, for example). That means you're going to need to do some form of specialization at runtime, so a JIT makes sense.
Re: DLVM: A modern compiler framework for neural network DSLs
#9Current tally of high-performance, deep-learning-oriented DSLs/IRs/compilers, in no particular order: - TensorComprehensions (Facebook): https://github.com/facebookresearch/TensorComprehensions - XLA (Google): https://www.tensorflow.org/performance/xla/ - taco (MIT): http://tensor-compiler.org/ - DLVM (UIUC): http://dlvm.org/ - nGraph (Intel): http://ngraph.nervanasys.com/docs/cpp/ - TVM (DMLC): https://github.com/dm…
As far as I know Tile/PlaidML (Vertex.AI) is the only DSL+compiler that's usable for real workloads across a variety of hardware. https://github.com/plaidml/plaidml
Tensorflow + XLA seems pretty usable. Also, it's generally good practice to note that you're a cofounder of Vertex.AI in discussions like this.
Re: DLVM: A modern compiler framework for neural network DSLs
#10Earlier quoted context omitted.
Well, the success of neural nets over the past few years has come through harnessing massive processing power. The problem is a lot of the programming can be low level and ad-hoc. I think the idea of the various DSLs is to allow the model to be compactly specified while having the programs go as fast as possible. A JIT may be one way to accomplish this.
Relying upon a JIT often means the ability to create things which preclude the use of static compilers, which means that accelerated hardware, like ours, cannot be used efficiently.
The kind of optimizations a static compiler might apply can be done by a JIT as well, with the added benefit of actually knowing what kind of workload is going to run. Most of deep learning is applying comparatively small computation graphs to very large arrays of numbers in parallel, so the overhead of compilation is only a small portion of the overall computation time. A smart JIT that decides on the optimal tiling pattern for the array dimensions observed at runtime and rewrites loops accordingly can easily pay for itself.