Supporting multiple backends (especially Jax) is nice! Makes experimenting/migrating between them so much more approachable. Any timeline on when can we expect support for distributed Jax training? The doc currently seems to indicate only TF is supported for distributed training.
Keras Core: Keras for TensorFlow, Jax, and PyTorch
41–50 of 73 posts
Re: Keras Core: Keras for TensorFlow, Jax, and PyTorch
#42I worked on the project, happy to answer any questions!
Just want to say I love Keras. Thank you for your work!
Re: Keras Core: Keras for TensorFlow, Jax, and PyTorch
#43I must admit i've never actually used keras but this is interesting to see how they are implementing it with Jax, definitely worth a reminder to dig into one of these days.
Re: Keras Core: Keras for TensorFlow, Jax, and PyTorch
#44Wouldn't a multi-framework wrapper be a subset of any supported framework's features common among all frameworks?
Additionally would it always be at least a step behind any framework depending on the wrapper's release cycle?
Re: Keras Core: Keras for TensorFlow, Jax, and PyTorch
#45Earlier quoted context omitted.
The same reason why you might want to use Keras if you use any of the other backends. They operate at different levels. Keras is a higher-level API. It means that you can prototype architectures quickly and you don't have to write a training loop. It's also really easy to extend. I currently use PyTorch Lightning to avoid having to write tonnes of boilerplate code, but I've been looking for a way to leave it for ages…
What’s so hard about writing a training loop?
nothing is. but to write a basic training loop with proper logging etc. from scratch every time you want to train a basic neural net classifier seems inefficient to me. There should be a framework for it where you can just plug in your model and your data and it trains it in a supervised fashion. That's what fast.ai or keras are doing.
Re: Keras Core: Keras for TensorFlow, Jax, and PyTorch
#46Earlier quoted context omitted.
Because sometimes you don’t want to write your own training loops, you just want a working method to train a model.
There are a lot of libraries for that. For example Pytorch Lightning, Accelerate are very mature
Sure, and Keras is another, very mature library which allows you to do this...
Re: Keras Core: Keras for TensorFlow, Jax, and PyTorch
#47Earlier quoted context omitted.
How so? You can use torch.nn.Sequential pretty much equivalently? https://pytorch.org/docs/stable/generated/torch.nn.Sequentia...
Huh, didn't realize base PyTorch had an equivalent Sequential API. The point about the better API overall still stands (notably including the actual training part, as base PyTorch requires you to implement your own loop)
Some people would say that this is an advantage of PyTorch, that it is very easy to write and play around with your own loop.
But if you don't want that, if you want to go a bit higher level w.r.t. training, there is e.g. PyTorch Lightning on top of PyTorch.
If you say the API is better in Keras, can you give any examples? They look kind of similar to me. Keras looks a bit more complicated, esp when looking at the internals (which I tend to do often, to better understand what's actually happening), which is a negative point.
Re: Keras Core: Keras for TensorFlow, Jax, and PyTorch
#48I think that is pretty cool - literally made me screen "Yes!" when I saw it and I don't do this for your everyday framework.
I think the beauty of keras was the perfect balance between simplicity/abstraction and flexibility. I moved to PyTorch eventually but one thing I always missed was this. And now, to have it leapfrog the current fragmentation and just achieve what seems to be a true multi-backend is pretty awesome.
Looking forward to the next steps!
Re: Keras Core: Keras for TensorFlow, Jax, and PyTorch
#49Earlier quoted context omitted.
> What's different now ? PyTorch adoption: back when Keras went hard into TensorFlow in 2018, both TF and PyTorch adoption were about the same with TF having a bit more popularity. Now, most of the papers and models released are PyTorch-first.
Yes I understand why they do the move (they want to attract pytorch user). What's the benefit for the user instead of directly using pytorch for example ? I see we can maybe use tpu by switching to jax etc... PS: sorry I'm a bit salty by my user experience of Keras.
Tensorflow has some advantages, like being able to use tf-lite for embedded devices. JAX is amazing on the TPU, which AFAIK pytorch doesn't have great support for.
I assume most people will still research in PyTorch, but then move it over the Keras for production models if they need multi-platform support.
Re: Keras Core: Keras for TensorFlow, Jax, and PyTorch
#50Earlier quoted context omitted.
What’s so hard about writing a training loop?
nothing is. but to write a basic training loop with proper logging etc. from scratch every time you want to train a basic neural net classifier seems inefficient to me. There should be a framework for it where you can just plug in your model and your data and it trains it in a supervised fashion. That's what fast.ai or keras are doing.
You only need to write a training loop function once. Then you can just pass to it a model, dataloader, etc, just like you would if you used a training loop written by someone else in Keras. The only difference is it would be hidden from you behind layers of wrappers and abstraction, making it harder to modify and debug.