I really like the idea of building AD directly into the language and compiler infrastructure itself, but the challenge of upstreaming it in a language built for very specific tasks makes me concerned. Is Latner just going to end up making a GSwift? Will it just be forked?
Differentiable Programming Mega-Proposal
61–70 of 78 posts
Re: Differentiable Programming Mega-Proposal
#62Ew, that’s really not something I would do at the language level...
Re: Differentiable Programming Mega-Proposal
#63How does this relate to all the work that Christ Lattner et al have been doing at Google with Swift, MLIR, etc.?[a] Is this... a separate, parallel, more encompassing proposal? Is there any coordination between these two groups? -- [a] https://www.youtube.com/watch?v=yCd3CzGSte8
I don't know if "Christ" Lattner was intentional (humorous) or an accident, but I chuckled.
Accident.
Re: Differentiable Programming Mega-Proposal
#64Earlier quoted context omitted.
There is a derivative, but since the function is non continuous, the derivative will likewise be messed up (but just around x=59). You really don’t want to be climbing a gradient around non continuous functions!
The function has a derivative by some notions of derivative. But the function's derivative can't be derived by an application of the chain rule and the know derivatives of primitive functions, which is what Algorithmic/automatic differentiation ultimately does (though it does this at run time, not compile time, since ordinary, symbolic differentiation explodes in memory for a complicated functions). Also: The continu…
I believe a source to source differentiator could deal with all these (where well defined of course), e.g.:
https://github.com/FluxML/Zygote.jl
julia> fs = Dict("sin" => sin, "cos" => cos, "tan" => tan);
julia> gradient(x -> fs[readline()](x), 1)
sin
0.5403023058681398Re: Differentiable Programming Mega-Proposal
#65I don't see why a well written library could not serve the same purpose. It seems like a lot of cruft. I doubt, for example, Python would ever consider adding this and it's the defacto language that would benefit the most from something like this - due to the existing tools and communities. It just seems so narrow and not at the same level of abstraction that languages typically sit at. I could see the language suppo…
I would counter that differentiable programming should perhaps rise to the level of baseline functionality that most languages should offer.
I think the applications for automatic differentiation and gradient optimization well exceed what we think of as ML and data science today.
Re: Differentiable Programming Mega-Proposal
#66Can "Differentiable Programming" be related to "Differentiable Privacy", or have we now one word (and acronym!) to describe two different things?
I think you meant "differential privacy", and no, it's not closely related to differentiable programming.
Re: Differentiable Programming Mega-Proposal
#67This will certainly help people who work on new deep learning theories and model architectures, but not so much the large crowd of deep learning practitioners.
In his excellent interview with Lex Fridman, Yann LeCun was critical of any approach to AI that was not differentiable, even constraint satisfaction, and other solid optimization techniques. In the context of scaling to very large problems or models with many billions of parameters, he is probably correct.
I have had problems with the Swift and TensorFlow code drops. Sometimes they work for me and sometime they don’t. So, very good technology but perhaps wait for it to mature. I read that some students for the fast.ai course using Swift have also had some setup difficulties.
EDIT: you might also want to look at Julia for differentiable programming and Julia with deep learning libraries like Flux is also a ‘turtles all the way down’ system, where unlike TensorFlow where the guts are implemented in C++, for Swift and Julia the entire stack can be implemented in a single language.
Re: Differentiable Programming Mega-Proposal
#68Earlier quoted context omitted.
Julia is in the same space.
Julia is great but it doesnt play in the domain of apps and servers
It does. I spent several evenings writing little bits of Julia code that do “non numeric” stuff like querying RDF data stores, text processing, etc. I think Julia is a reasonable general purpose language.
Re: Differentiable Programming Mega-Proposal
#69Earlier quoted context omitted.
I still don't get it. Why can't I use a debugger to step through derivatives when autodiff is implemented as a library?
Reverse mode autodiff is best implemented with a non-local transformation of the program: First you run the original operations forward; then you run corresponding operations in reverse order. You can do this with a library by implementing "number" types that, as a side effect of arithmetic operations, record those operations onto a "tape", so that corresponding (different) operations can be played back later in reve…
Just complementing the other reply, there is a small article about how you can implement a source to source AD in Julia [1]. Basically Julia has a special type of macro called generated function [2] which instead of executing during the AST lowering phase (when the compiler still didn't evaluate the symbols) it executes during the final step of compilation (when type inference already ran and you have all the exact types), and in that function you can return either the AST or Julia's SSA IR directly (which is good for AD since it closely resembles the execution graph since it avoids mutability). And you can also inspect the IR of any function call and manipulate it within the language [3], so you can recursively create the tape entirely at compile time.
[1] http://blog.rogerluo.me/2019/07/27/yassad/
[2] https://docs.julialang.org/en/v1/manual/metaprogramming/inde...
[3] https://mikeinnes.github.io/IRTools.jl/latest/#Evaluating-IR...
Re: Differentiable Programming Mega-Proposal
#70How does this compare to Jax? https://github.com/google/jax Why do this in the language instead of a library?
Probably the main justification is that the analysis and transformation steps needed to compute the vjp and jvp pullbacks of a function (which correspond to reverse- and forward-mode automatic differentiation) require enough of the other machinery of a compiler that they are best done WITHIN a compiler. Then other things become quite natural, too, like producing the tangent vector versions of data structures like tup…
jax does have a just in time compiler that lets you compile your python functions to XLA-optimized kernels (through llvmlite under the hood?).
The fact that you can jit compile and gain the benefits of "doing this within the compiler" is one of its main selling points.