Growing a Compiler: Getting to Machine Learning from a General Purpose Compiler
1–10 of 40 posts
Re: Growing a Compiler: Getting to Machine Learning from a General Purpose Compiler
#2wat
Re: Growing a Compiler: Getting to Machine Learning from a General Purpose Compiler
#3wat
This is a talk we gave at the "Compilers for Machine Learning" workshop at CGO this past Sunday. Obviously the slides are missing some of the context of the presentation, but the core argument is that one shouldn't build compilers for machine learning, but rather build general purpose compilers that are flexible and extensible. From there, you can easily get all the same benefits as dedicated machine learning compilers without building a huge monolith. The rest of the slides highlight some of the work done in the Julia community to work in this direction.
Re: Growing a Compiler: Getting to Machine Learning from a General Purpose Compiler
#4wat
This is a talk we gave at the "Compilers for Machine Learning" workshop at CGO this past Sunday. Obviously the slides are missing some of the context of the presentation, but the core argument is that one shouldn't build compilers for machine learning, but rather build general purpose compilers that are flexible and extensible. From there, you can easily get all the same benefits as dedicated machine learning compile…
Adding to the context, we wrote up a little post about the C4ML workshop: https://juliacomputing.com/blog/2019/02/19/growing-a-compile...
Reproducing it here: The Compilers for Machine Learning workshop was recently held at CGO 2019. Since compiler techniques affect a large part of the machine learning stack, this workshop aimed to highlight research that incorporates compiler techniques and algorithms in optimizing machine learning workloads. The workshop included talks from various projects - Julia (Julia Computing), TVM (UW), Glow (Facebook), XLA (Google), nGraph (Intel), TensorRT (Nvidia), and the soon to release MLIR (Google).
Our talk introduced the abstractions in the Julia language and the kind of compiler transforms involved in implementing them. We then had a deep dive into dynamic semantics + static analysis - our JAOT (Just-Ahead-Of-Time) analysis. Building on these capabilities, the Zygote system implements automatic differentiation, effectively treating it as a compiler problem, giving us differentiable programming for free. Finally, compiler backends for GPUs and TPUs give us high performance execution. All this comes together beautifully in Neural ODEs, which we had to show off as our first slide!
Re: Growing a Compiler: Getting to Machine Learning from a General Purpose Compiler
#5wat
[deleted]
Re: Growing a Compiler: Getting to Machine Learning from a General Purpose Compiler
#6Earlier quoted context omitted.
This is a talk we gave at the "Compilers for Machine Learning" workshop at CGO this past Sunday. Obviously the slides are missing some of the context of the presentation, but the core argument is that one shouldn't build compilers for machine learning, but rather build general purpose compilers that are flexible and extensible. From there, you can easily get all the same benefits as dedicated machine learning compile…
Adding to the context, we wrote up a little post about the C4ML workshop: https://juliacomputing.com/blog/2019/02/19/growing-a-compile... Reproducing it here: The Compilers for Machine Learning workshop was recently held at CGO 2019. Since compiler techniques affect a large part of the machine learning stack, this workshop aimed to highlight research that incorporates compiler techniques and algorithms in optimizing…
Also, since Yann LeCun said yesterday that Deep Learning needs a new language, this has become a major topic of interest.
https://venturebeat.com/2019/02/18/facebooks-chief-ai-scient...
Of course, our view is that Julia is one such language that people should consider seriously. The talk linked here is a peek under the hood and shows that differentiable programming in Julia is not a special add-on, but something that fits naturally within the language.
Re: Growing a Compiler: Getting to Machine Learning from a General Purpose Compiler
#7wat
This is a talk we gave at the "Compilers for Machine Learning" workshop at CGO this past Sunday. Obviously the slides are missing some of the context of the presentation, but the core argument is that one shouldn't build compilers for machine learning, but rather build general purpose compilers that are flexible and extensible. From there, you can easily get all the same benefits as dedicated machine learning compile…
How ... whatever was the reason for thinking that you needed a ML-specific compiler?
Do people also think you need a timecard-tracking-specific compiler?
Re: Growing a Compiler: Getting to Machine Learning from a General Purpose Compiler
#8Earlier quoted context omitted.
Adding to the context, we wrote up a little post about the C4ML workshop: https://juliacomputing.com/blog/2019/02/19/growing-a-compile... Reproducing it here: The Compilers for Machine Learning workshop was recently held at CGO 2019. Since compiler techniques affect a large part of the machine learning stack, this workshop aimed to highlight research that incorporates compiler techniques and algorithms in optimizing…
Also, since Yann LeCun said yesterday that Deep Learning needs a new language, this has become a major topic of interest. https://venturebeat.com/2019/02/18/facebooks-chief-ai-scient... Of course, our view is that Julia is one such language that people should consider seriously. The talk linked here is a peek under the hood and shows that differentiable programming in Julia is not a special add-on, but something that…
A long time ago, Yann LeCun wrote Lush, which was a numerically-focused Lisp dialect, with a focus on C interop. It might be one of Julia's closest sibling.
> Lush is an object-oriented programming language designed for researchers, experimenters, and engineers interested in large-scale numerical and graphic applications. Lush is designed to be used in situations where one would want to combine the flexibility of a high-level, weakly-typed interpreted language, with the efficiency of a strongly-typed, natively-compiled language, and with the easy integration of code written in C, C++, or other languages.
http://lush.sourceforge.net/index.html
Is there more detail about his proposal? It must be well thought-out.
Re: Growing a Compiler: Getting to Machine Learning from a General Purpose Compiler
#9Earlier quoted context omitted.
This is a talk we gave at the "Compilers for Machine Learning" workshop at CGO this past Sunday. Obviously the slides are missing some of the context of the presentation, but the core argument is that one shouldn't build compilers for machine learning, but rather build general purpose compilers that are flexible and extensible. From there, you can easily get all the same benefits as dedicated machine learning compile…
How ... whatever was the reason for thinking that you needed a ML-specific compiler? Do people also think you need a timecard-tracking-specific compiler?
Well, I do think there are some very real shortcomings that current general purpose compilers have when applied to machine learning (aggrevated by the fact that lots of machine learning code is written in languages with poor compiler support), so lots people looked at that, wrote small optimizers and got good performance. But then it turned out that researchers wanted to do more and more things in their ML models and these small optimizers are turning into full-blown compilers, with not a lot of thought about whether that is truly the correct thing to do.
Re: Growing a Compiler: Getting to Machine Learning from a General Purpose Compiler
#10[deleted]