71–80 of 111 posts
From what I see, still no 3.11 support — Same for Tensorflow which won't ship before Q1 23.
It's not a huge deal, as the speed improvements in 3.11 likely wouldn't trickle down at the core PyTorch level.
How is PyTorch compares to JAX and its stack?
I find that Jax tends to result in messy code unless you build good abstractions. I personally don't like Flax and Haiku, I prefer stax and Equinox as they are more transparent on what is happening, feel a lot less like magic, and more pythonic (explicit is better than implicit etc). PyTorch is far more friendly for deep learning stuff, but sometimes all you want is pure numerical computations that can be vmapped acr…
You probably were not doing anything wrong. I spent a lot of time trying to be clever in order to parallelize things like this and it just wasn't possible without doing CUDA extensions. But it is now! PyTorch now has vmap through functorch and it works.
Earlier quoted context omitted.
Ok, is there any way to trim down the amount of code used without reducing the performance of my particular application, and my particular machine? I have the feeling that it's an all-or-nothing proposition. Either you have a simple CPU-only algorithm, or you have several gigabytes of libraries you don't really need. Also, in some applications I would be willing to give up 10% of performance if I could reclaim 90% of…
CuDNN is only for Nvidia GPUs, and those machines generally have decent sized disks and decent network connections so no nobody cares about a few GBs of libraries. There are alternatives to using CuDNN with much smaller binary size. Maybe they can match or beat it or maybe not, depending on your model and hardware. But you'll have to do your own work to switch to them, since most people are happy enough with CuDNN fo…
Yes, I agree about the driver situation.
Earlier quoted context omitted.
At least prior to this announcement: JAX was much faster than PyTorch for differentiable physics. (Better JIT compiler; reduced Python-level overhead.) E.g for numerical ODE simulation, I've found that Diffrax ( https://github.com/patrick-kidger/diffrax ) is ~100 times faster than torchdiffeq on the forward pass. The backward pass is much closer, and for this Diffrax is about 1.5 times faster. It remains to be seen h…
If you care about performance of differential physics you shouldn't use python. Diffrax is almost OKish, but is missing a ton of features (e.g. good stiff solvers, arbitrary precision support, events for anything other than stopping the simulation, ability to control the linear solve which are needed for large problems). For simple cases it can come close to the C++/Julia solvers, but for anything complicated, you ei…
This definitely isn't true. On any benchmark I've tried, JAX and Julia basically match each other. Usually I find JAX to be a bit faster, but that might just be that I'm a bit more skilled at optimising that framework.
Anyway I'm not going to try and debunk things point-by-point, I'd rather avoid yet another unpleasant Julia flame-war.
Earlier quoted context omitted.
CuDNN is only for Nvidia GPUs, and those machines generally have decent sized disks and decent network connections so no nobody cares about a few GBs of libraries. There are alternatives to using CuDNN with much smaller binary size. Maybe they can match or beat it or maybe not, depending on your model and hardware. But you'll have to do your own work to switch to them, since most people are happy enough with CuDNN fo…
It's not just disk size. Also memory size, and loading speed. Yes, I agree about the driver situation.
A big lesson I learned from PyTorch vs other frameworks is that productivity trumps incremental performance improvement. Both Caffe and MXNet marketed themselves for being fast, yet apparently being faster here and here by some percentage simply didn't matter that much. On the other hand, once we make a system work and make it popular, the community will close the performance gap sooner than competitors expect. Anoth…
What I really find interesting here is that PyTorch, a library maintained by Facebook, is winning the marketshare and mindshare due to clean API, whereas Tensorflow, maintained by Google, is losing due to inferior API. In general, Google as a company emphasizes code quality and best practices far more than Facebook. But the story was reversed here.
A big lesson I learned from PyTorch vs other frameworks is that productivity trumps incremental performance improvement. Both Caffe and MXNet marketed themselves for being fast, yet apparently being faster here and here by some percentage simply didn't matter that much. On the other hand, once we make a system work and make it popular, the community will close the performance gap sooner than competitors expect. Anoth…
What I really find interesting here is that PyTorch, a library maintained by Facebook, is winning the marketshare and mindshare due to clean API, whereas Tensorflow, maintained by Google, is losing due to inferior API. In general, Google as a company emphasizes code quality and best practices far more than Facebook. But the story was reversed here.
Earlier quoted context omitted.
I find that Jax tends to result in messy code unless you build good abstractions. I personally don't like Flax and Haiku, I prefer stax and Equinox as they are more transparent on what is happening, feel a lot less like magic, and more pythonic (explicit is better than implicit etc). PyTorch is far more friendly for deep learning stuff, but sometimes all you want is pure numerical computations that can be vmapped acr…
> Personal Example: I needed to sample a bunch of datapoints, make distributions out of them, sample, and then compute the density of each sample across distributions. Doing this with pytorch was rather slow, I was probably doing something wrong with vectorization and broadcasting, but I didn't have the time to figure it out. You probably were not doing anything wrong. I spent a lot of time trying to be clever in ord…
A lot of this is down to driver availability and software stacks...but is all of it? Game engines/engineers seem to be able to be productive on a wide variety of GPU hardware, why do so many ML libraries just not provide any support at all? Sure 75% of potential performance is less satisfying than 100%, but it's also infinitely better than 0%. How come every ML library doesn't at least have an OpenCL fallback or the like?
Earlier quoted context omitted.
> Personal Example: I needed to sample a bunch of datapoints, make distributions out of them, sample, and then compute the density of each sample across distributions. Doing this with pytorch was rather slow, I was probably doing something wrong with vectorization and broadcasting, but I didn't have the time to figure it out. You probably were not doing anything wrong. I spent a lot of time trying to be clever in ord…
I found that there still some limitations with functorch's vmap, but I can't recall what it was.