I really don't understand why they didn't go with Julia, it fits the purpose much better than Swift, already having a ML ecosystem and having great interop with Python, C, C++, R and Matlab.
Heck JAX, where a lot of TF refugees are going, is pretty similar to Zygote
Swift for TensorFlow – A system for deep learning and differentiable computing
101–110 of 142 posts
Re: Swift for TensorFlow – A system for deep learning and differentiable computing
#102Earlier quoted context omitted.
I'm skeptical of JAX. It feels good right now, but when the first TF beta version came out it was very much like that too - clean, simple, minimal, and just a better version of Theano. Then the "crossing the chasm" effort started and everyone at Google wanted to be part of it, making TF the big complex mess it is today. It's a great example of Conway's Law. I'm not convinced the same won't happen to JAX as it catches…
> I'm not convinced the same won't happen to JAX And now there are already multiple NN libraries for JAX from Google...
There are a bunch of frameworks built on top of Pytorch too (fastAI, lighting, torchbearer, ignite...), I don't see why this should be a problem (or at least a problem to JAX but not to Pytorch)
Re: Swift for TensorFlow – A system for deep learning and differentiable computing
#103I'm not sure this is really going to take off, it seems that most people who are abandoning TF are moving to Jax or pytorch. My own experience with Jax is that it is much easier to use then TF, just an all round more pleasant experience. It would be interesting to try this, but at this point I'm not really willing to learn 'yet another deep learning framework' and the extreme anti-user problems that TF had make me lo…
I have just started hearing about Jax. But it seems to be a low level library that Tensorflow uses right ? The latest release of Tensorflow probability uses JAX under the hood. So what do you mean when you say you're moving to JAX versus Tensorflow
In your first sentence you're mistaking JAX and XLA
XLA: Accelerated Linear Algebra, I guess it's kind of a backend/compiler that optimizes Linear Algebra/Deep Learning calculations with some very interesting techniques, among them fusing kernels
JAX: In some sense syntax sugar over XLA, but a better way of describing it is Composable transformations + Numpy + some Scipy. The composable transformations allow you to take derivatives (be them single, multi or vector valued functions and also higher order derivatives), JIT a function (which is them compiled to XLA), 2 forms of parallelism (vmap and pmap) and others, while being compatible with one another and with both TPUs, GPUs and CPUs
Re: Swift for TensorFlow – A system for deep learning and differentiable computing
#104Earlier quoted context omitted.
I know where you're coming from, but TF in my opinion was very user-hostile even on arrival. I can't tell you how much hair-pulling I did over tf.conds, tf.while_loops and the whole gather / scatter paradigm for simple indexing into arrays. I really think the people working on it wanted users to write TF code in a certain, particular way and made it really difficult to use it in other ways. Just thinking back on that…
(googler) Not quite sure why you picked those particular examples... JAX also requires usage of lax.cond, lax.while_loop, and ops.segment_sum. Only gather has been improved with slice notation support. IMO, TF has landed on a pretty nice solution to cond/while_loop via AutoGraph.
While jax has those operations you don't always need them, it depends on what transformations you want to do (JIT or grad) and they have been working on making normal control structures compatible with all transformations
Re: Swift for TensorFlow – A system for deep learning and differentiable computing
#105Earlier quoted context omitted.
> I'm not convinced the same won't happen to JAX And now there are already multiple NN libraries for JAX from Google...
There are a bunch of frameworks built on top of Pytorch too (fastAI, lighting, torchbearer, ignite...), I don't see why this should be a problem (or at least a problem to JAX but not to Pytorch)
IMO, this is not a fair comparison because Pytorch spans a larger amount of abstraction than jax (I don't quite know how to explain it other than "spans a larger amount of abstraction").
You can do much of the jax stuff in pytorch, you can't do the high level nn.LSTM stuff in jax, you have to use like flax or objax or something.
Re: Swift for TensorFlow – A system for deep learning and differentiable computing
#106Careful, folks. S4TF is pretty much dead on arrival. It was pushed aggressively by Chris Lattner (for obvious reasons) but he left Google a while ago and since then most internal users lost interest. There's nothing in Swift that's inherently suitable for ML and building the ecosystem is a ton of work; without all the political pushing, it went nowhere and is close to a "semi-abandoned research project" phase.
If true, it does not surprise me. While there is a lot of language war going on in the ML ecosystem, I never really heard anyone using, planning to use or waiting for swift for TensorFlow.
It might be a nice language (never used it), but there are other contenders with more engineers/scientists support like Rust and Julia, for which the advantages were clearer.
Finally, the whole ordeal got a very bad look from having its main proponent being the creator of Swift instead of the actual community pushing for it.
Re: Swift for TensorFlow – A system for deep learning and differentiable computing
#107Earlier quoted context omitted.
I can't speak for Apple, of course, but some indications of their seriousness are there. SwiftNIO is a cross-platform asynchronous event-driven network application framework for rapid development of maintainable high performance protocol servers & clients. ( https://github.com/apple/swift-nio ) Distributed Membership Protocol implementations in Swift: https://github.com/apple/swift-cluster-membership Docker Official…
Still waiting for the day when import Glibc isn't a thing on Swift examples.
Well, you can help make it come sooner… wink
Re: Swift for TensorFlow – A system for deep learning and differentiable computing
#108Careful, folks. S4TF is pretty much dead on arrival. It was pushed aggressively by Chris Lattner (for obvious reasons) but he left Google a while ago and since then most internal users lost interest. There's nothing in Swift that's inherently suitable for ML and building the ecosystem is a ton of work; without all the political pushing, it went nowhere and is close to a "semi-abandoned research project" phase.
If true, it does not surprise me. While there is a lot of language war going on in the ML ecosystem, I never really heard anyone using, planning to use or waiting for swift for TensorFlow. It might be a nice language (never used it), but there are other contenders with more engineers/scientists support like Rust and Julia, for which the advantages were clearer. Finally, the whole ordeal got a very bad look from havin…
Precisely because Chris Latter had the idea of replacing everything from C to Javascript with Swift.
I didn't like that idea then, I still dont like it now. But the idea in itself is very Apple (ish).
Re: Swift for TensorFlow – A system for deep learning and differentiable computing
#109Earlier quoted context omitted.
Sure. I guess then you shouldn’t even use the word “dead”, as plenty of people have lost close relatives which is a pretty traumatic event.
I think the difference is that the term "Dead" is used in a lot of different contexts. E.g. Battery is dead. Stillborn is not, and it mainly associated used in one very traumatic context.
How about “miscarriage of justice”?
Re: Swift for TensorFlow – A system for deep learning and differentiable computing
#110Earlier quoted context omitted.
Why is that? It conveys the DOA meaning pretty well.
Generally, your metaphors should not rely on comparison to a pretty traumatic event that has happened to quite a few people, many of whom might be around you without you knowing.
DOA is a metaphor too, literally referring to people arriving at the ER too late.