Swift for TensorFlow – A system for deep learning and differentiable computing

Post by MiroF » Mon, Sep 21, 2020, 4:08 PM UTC

dklend122 wrote: post #117

mkolodny wrote: post #82

Earlier quoted context omitted.

Speed, too. For PyTorch to train models and run inference quickly, your Python code gets translated to C++/CUDA. Part of the idea with S4TF is to be able to write ML code in a single, fast language.

Well, it's not happening in swift yet. S4TF still requires either c cuda kernels or XLA. Julia on the other hand has JIT GPU codegen and its CPU codegen has been benchmarked to beat openblas

Post by cgs1019 » Mon, Sep 21, 2020, 7:12 PM UTC

sandGorgon wrote: post #112

joaogui1 wrote: post #103

Earlier quoted context omitted.

In your first sentence you're mistaking JAX and XLA XLA: Accelerated Linear Algebra, I guess it's kind of a backend/compiler that optimizes Linear Algebra/Deep Learning calculations with some very interesting techniques, among them fusing kernels JAX: In some sense syntax sugar over XLA, but a better way of describing it is Composable transformations + Numpy + some Scipy. The composable transformations allow you to t…

im not mistaking the articles around it - check this out: https://www.tensorflow.org/probability/examples/TensorFlow_P... "TensorFlow Probability (TFP) is a library for probabilistic reasoning and statistical analysis that now works on JAX! For those not familiar, JAX is a library for accelerated numerical computing based on composable function transformations. We have ported a lot of TFP's most useful functionality…

Post by yunohn » Mon, Sep 21, 2020, 8:22 PM UTC

c16a wrote: post #93

yunohn wrote: post #30

Earlier quoted context omitted.

I agree on the hardware part. MacOS only supports AMD GPUs which are unsupported by Tensor flow...

Tensorflow can work with ROCM which is equivalent to Nvidia's CUDA.

Post by amkkma » Mon, Sep 21, 2020, 9:04 PM UTC

yunohn wrote: post #126

c16a wrote: post #93

Earlier quoted context omitted.

Tensorflow can work with ROCM which is equivalent to Nvidia's CUDA.

There is no official support, only AMD's custom TF branch. Further ROCM itself works only on Linux. I believe most people writing Swift are using MacOS.

Post by mattjjatgoogle » Mon, Sep 21, 2020, 10:37 PM UTC

MiroF wrote: post #69

bodono wrote: post #39

Earlier quoted context omitted.

I know where you're coming from, but TF in my opinion was very user-hostile even on arrival. I can't tell you how much hair-pulling I did over tf.conds, tf.while_loops and the whole gather / scatter paradigm for simple indexing into arrays. I really think the people working on it wanted users to write TF code in a certain, particular way and made it really difficult to use it in other ways. Just thinking back on that…

> tf.conds, tf.while_loops and the whole gather / scatter paradigm I'm ill-informed - but isn't that exactly what lax is?

  In [1]: from jax import grad
  
  In [2]: def f(x):
     ...:     if x > 0:
     ...:         return 3. * x ** 2
     ...:     else:
     ...:         return 5. * x ** 3
     ...:
  
  In [3]: grad(f)(1.)
  Out[3]: DeviceArray(6., dtype=float32)
  
  In [4]: grad(f)(-1.)
  Out[4]: DeviceArray(15., dtype=float32)