Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Swift for TensorFlow – A system for deep learning and differentiable computing

tensorflow.org

Post reply on HN
Print view View original

131–140 of 142 posts

Re: Swift for TensorFlow – A system for deep learning and differentiable computing

#131

Post by p1esk »

mattjjatgoogle wrote: post #130
MiroF wrote: post #69

Earlier quoted context omitted.

> tf.conds, tf.while_loops and the whole gather / scatter paradigm I'm ill-informed - but isn't that exactly what lax is?

The difference is that in TF1 you had to use tf.cond, tf.while_loop etc for differentiable control flow. In JAX you can differentiate Python control flow directly, e.g.: In [1]: from jax import grad In [2]: def f(x): ...: if x > 0: ...: return 3. * x ** 2 ...: else: ...: return 5. * x ** 3 ...: In [3]: grad(f)(1.) Out[3]: DeviceArray(6., dtype=float32) In [4]: grad(f)(-1.) Out[4]: DeviceArray(15., dtype=float32) In t…

What would you say the main advantage of Jax is over Pytorch?

Re: Swift for TensorFlow – A system for deep learning and differentiable computing

#132

Post by MiroF »

ddragon wrote: post #123
MiroF wrote: post #122

Earlier quoted context omitted.

> CPU codegen has been benchmarked to beat openblas Source on this claim?

He probably means Tullio.jl (which also seems to integrate with Julia's source to source differentiation library Zygote.jl, the main competitor to Swift for Tensorflow): https://discourse.julialang.org/t/realistically-how-close-is... https://github.com/mcabbott/Tullio.jl Regardless if it can consistently beat Fortran/BLAS in every area, in general JIT languages have more opportunities for optimizations than AoT langu…

I'm surprised re: blas - that is closer than I thought! Still a huge gap on the GPU, but still impressive.

> in general JIT languages have more opportunities for optimizations than AoT languages

I'm not sure I agree with this. I would say the opportunities for any non-GC mature AoT language (C++, Rust, etc.) is going to be pretty much the same, since you can just attach a JIT to most AoT langs.

Re: Swift for TensorFlow – A system for deep learning and differentiable computing

#133

Post by amkkma »

MiroF wrote: post #132
ddragon wrote: post #123

Earlier quoted context omitted.

He probably means Tullio.jl (which also seems to integrate with Julia's source to source differentiation library Zygote.jl, the main competitor to Swift for Tensorflow): https://discourse.julialang.org/t/realistically-how-close-is... https://github.com/mcabbott/Tullio.jl Regardless if it can consistently beat Fortran/BLAS in every area, in general JIT languages have more opportunities for optimizations than AoT langu…

I'm surprised re: blas - that is closer than I thought! Still a huge gap on the GPU, but still impressive. > in general JIT languages have more opportunities for optimizations than AoT languages I'm not sure I agree with this. I would say the opportunities for any non-GC mature AoT language (C++, Rust, etc.) is going to be pretty much the same, since you can just attach a JIT to most AoT langs.

The GPU gap is only if written in the high level index or loop style. There is little to no gap if done either using array abstractions (broadcast, map etc) or at a level similar to Cuda C (though with nicer Julia abstractions and syntax): https://juliagpu.org/cuda/

The Julialab at MIT is working on making the higher level codegen faster

Re: Swift for TensorFlow – A system for deep learning and differentiable computing

#134

Post by ddragon »

MiroF wrote: post #132
ddragon wrote: post #123

Earlier quoted context omitted.

He probably means Tullio.jl (which also seems to integrate with Julia's source to source differentiation library Zygote.jl, the main competitor to Swift for Tensorflow): https://discourse.julialang.org/t/realistically-how-close-is... https://github.com/mcabbott/Tullio.jl Regardless if it can consistently beat Fortran/BLAS in every area, in general JIT languages have more opportunities for optimizations than AoT langu…

I'm surprised re: blas - that is closer than I thought! Still a huge gap on the GPU, but still impressive. > in general JIT languages have more opportunities for optimizations than AoT languages I'm not sure I agree with this. I would say the opportunities for any non-GC mature AoT language (C++, Rust, etc.) is going to be pretty much the same, since you can just attach a JIT to most AoT langs.

Sure, if you give static language a JIT they'll be able to get the advantages of having JIT, though language semantics still matter. A language built for JITs like Julia or Common Lisp have native ways of interfacing with the compiler, and programs are built without worry of exponential explosion of implementations during method monomorphization (as you'll only compile the optimal versions that you'll actually use, based on runtime information, without having to be pessimist as any overspecialization can be fixed on demand). AoT languages would probably need a compiler pragma or type similar to a dynamic boxing but for delayed monomorphization/compilation for methods when you want to avoid compiling all paths AoT (which might be a way to allow for example tensor specialization on sizes, similar to StaticArrays on Julia).

Re: Swift for TensorFlow – A system for deep learning and differentiable computing

#135

Post by MiroF »

ddragon wrote: post #134
MiroF wrote: post #132

Earlier quoted context omitted.

I'm surprised re: blas - that is closer than I thought! Still a huge gap on the GPU, but still impressive. > in general JIT languages have more opportunities for optimizations than AoT languages I'm not sure I agree with this. I would say the opportunities for any non-GC mature AoT language (C++, Rust, etc.) is going to be pretty much the same, since you can just attach a JIT to most AoT langs.

Sure, if you give static language a JIT they'll be able to get the advantages of having JIT, though language semantics still matter. A language built for JITs like Julia or Common Lisp have native ways of interfacing with the compiler, and programs are built without worry of exponential explosion of implementations during method monomorphization (as you'll only compile the optimal versions that you'll actually use, b…

I don't quite follow.

I am not too experienced with Julia, but my understanding was that it uses LLVM to jit itself. Since the LLVM jit compiler is also an API available to C++, anything that can be done in Julia can be done with jit to LLVM api in C++.

Then you just compile the methods that you'll actually use with LLVM right before using them.

Re: Swift for TensorFlow – A system for deep learning and differentiable computing

#136

Post by MiroF »

amkkma wrote: post #133
MiroF wrote: post #132

Earlier quoted context omitted.

I'm surprised re: blas - that is closer than I thought! Still a huge gap on the GPU, but still impressive. > in general JIT languages have more opportunities for optimizations than AoT languages I'm not sure I agree with this. I would say the opportunities for any non-GC mature AoT language (C++, Rust, etc.) is going to be pretty much the same, since you can just attach a JIT to most AoT langs.

The GPU gap is only if written in the high level index or loop style. There is little to no gap if done either using array abstractions (broadcast, map etc) or at a level similar to Cuda C (though with nicer Julia abstractions and syntax): https://juliagpu.org/cuda/ The Julialab at MIT is working on making the higher level codegen faster

I guess that makes sense to me.. you can just automatically convert the C in BLAS to Julia and then if they're both being converted to llvm ir by clang anyways than i guess it'll be about as fast!

Re: Swift for TensorFlow – A system for deep learning and differentiable computing

#137

Post by sandGorgon »

cgs1019 wrote: post #124
sandGorgon wrote: post #112

Earlier quoted context omitted.

im not mistaking the articles around it - check this out: https://www.tensorflow.org/probability/examples/TensorFlow_P... "TensorFlow Probability (TFP) is a library for probabilistic reasoning and statistical analysis that now works on JAX! For those not familiar, JAX is a library for accelerated numerical computing based on composable function transformations. We have ported a lot of TFP's most useful functionality…

Hi, tech lead for TFP here. The wording here was unclear -- sorry! We're fixing it presently. We are not migrating away from TF; far from it! The change here was to interoperate with TF and JAX (and numpy!), by way of some rewrite trickery under the hood. Essentially, we wrote a translation layer that implements the TF API surface (or, the parts we actually use) in terms of numpy & JAX primitives [1]. This lets us le…

hey thanks for the clarification.

here's what everybody is puzzled on: it looks like the layers going forward are JAX -> Tensorflow -> Keras.

and we are seeing people moving to JAX directly. So this is ending up like a Flutter vs Kotlin issue (also within Google).

Do you envision JAX being low level .. and the high level tensorflow keras interface being the most usable api ?

Re: Swift for TensorFlow – A system for deep learning and differentiable computing

#138

Post by shele »

MiroF wrote: post #135
ddragon wrote: post #134

Earlier quoted context omitted.

Sure, if you give static language a JIT they'll be able to get the advantages of having JIT, though language semantics still matter. A language built for JITs like Julia or Common Lisp have native ways of interfacing with the compiler, and programs are built without worry of exponential explosion of implementations during method monomorphization (as you'll only compile the optimal versions that you'll actually use, b…

I don't quite follow. I am not too experienced with Julia, but my understanding was that it uses LLVM to jit itself. Since the LLVM jit compiler is also an API available to C++, anything that can be done in Julia can be done with jit to LLVM api in C++. Then you just compile the methods that you'll actually use with LLVM right before using them.

The crux is that which are "the methods that you'll actually use" is very difficult to answer. A lot of effort is put into this on the Julia "Package compiler" and "Compiler" projects.

Re: Swift for TensorFlow – A system for deep learning and differentiable computing

#139

Post by ddragon »

MiroF wrote: post #135
ddragon wrote: post #134

Earlier quoted context omitted.

Sure, if you give static language a JIT they'll be able to get the advantages of having JIT, though language semantics still matter. A language built for JITs like Julia or Common Lisp have native ways of interfacing with the compiler, and programs are built without worry of exponential explosion of implementations during method monomorphization (as you'll only compile the optimal versions that you'll actually use, b…

I don't quite follow. I am not too experienced with Julia, but my understanding was that it uses LLVM to jit itself. Since the LLVM jit compiler is also an API available to C++, anything that can be done in Julia can be done with jit to LLVM api in C++. Then you just compile the methods that you'll actually use with LLVM right before using them.

Sorry, you're right in that Julia is written in C/C++, so everything Julia does can be solved in those by writing a language (like Julia itself, and not unlike Tensorflow original interface) and compiling it on demand and finding a way to eval the new code and recover the results. I was talking along the ways of how to make it sort of convenient (at least viable to implent unlike the former), as an extension to the C++ compiler itself where you can just tell the compiler what stays AoT and what is JIT'd but otherwise keep the same C++ syntax.

Not to mention if you want to reimplement Julia's logic in C++ you'll have to develop it's sophisticated type inference, since Julia compiler is so aggressive that it will compile at once entire blocks of program (the entire program if it can) as long as it can infer what types are used downstream, which is why it can compete with AoT compiled languages (it's basicaly a "Just Ahead of Time Compiler")

Re: Swift for TensorFlow – A system for deep learning and differentiable computing

#140

Post by amkkma »

MiroF wrote: post #136
amkkma wrote: post #133

Earlier quoted context omitted.

The GPU gap is only if written in the high level index or loop style. There is little to no gap if done either using array abstractions (broadcast, map etc) or at a level similar to Cuda C (though with nicer Julia abstractions and syntax): https://juliagpu.org/cuda/ The Julialab at MIT is working on making the higher level codegen faster

I guess that makes sense to me.. you can just automatically convert the C in BLAS to Julia and then if they're both being converted to llvm ir by clang anyways than i guess it'll be about as fast!

That's not at all what Julia is doing. It's much more sophisticated in that it has very low level intrinsic primitives that can compose and it optimizes the IR to make it fast and then compiles it to CUDA. These all map to Julia constructs.
Post reply on HN