Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

PyTorch 2.0

pytorch.org

Post reply on HN
Print view View original

51–60 of 111 posts

Re: PyTorch 2.0

#51

Post by patrickkidger »

uoaei wrote: post #32
whimsicalism wrote: post #17

Earlier quoted context omitted.

I was reading this and thinking it was a pretty terrible answer - glad it is just generated by an AI and not you personally so I'm not insulting you. JAX is basically numpy on steroids and lets you do a lot of non-standard things (like a differentiable physics simulation or something) that would be harder with Pytorch. They are both "high-performance." Pytorch is more geared towards traditional deep learning and has…

Can someone comment more on what makes JAX that much better for differentiable simulations than PyTorch? I'm working on a new module for work and none of my colleagues have much experience developing ML per se. I'm trying to decide whether to force their hand by implementing v1 in PyTorch or JAX and differentiable physics simulations is a likely future use case. Why is PyTorch harder?

At least prior to this announcement: JAX was much faster than PyTorch for differentiable physics. (Better JIT compiler; reduced Python-level overhead.)

E.g for numerical ODE simulation, I've found that Diffrax (https://github.com/patrick-kidger/diffrax) is ~100 times faster than torchdiffeq on the forward pass. The backward pass is much closer, and for this Diffrax is about 1.5 times faster.

It remains to be seen how PyTorch 2.0 will compare, of course!

Right now my job is actually building out the scientific computing ecosystem in JAX, so feel free to ping me with any other questions.

Re: PyTorch 2.0

#53

Post by la_fayette »

SleekEagle wrote: post #45
hintymad wrote: post #25

A big lesson I learned from PyTorch vs other frameworks is that productivity trumps incremental performance improvement. Both Caffe and MXNet marketed themselves for being fast, yet apparently being faster here and here by some percentage simply didn't matter that much. On the other hand, once we make a system work and make it popular, the community will close the performance gap sooner than competitors expect. Anoth…

Exactly, especially in the age of ridiculously rapid development that we have found ourselves in over the past few years. This is exactly why TensorFlow is dying

:) it still is impossible to bring a cnn rnn network in pytorch to mobile, which works fine with tflite...

Re: PyTorch 2.0

#54

Post by modeless »

claytonjy wrote: post #50
modeless wrote: post #44

Earlier quoted context omitted.

The complexity of deep learning algorithms is low but the complexity of the hardware is high. The problem solved by these gigabytes of libraries is getting peak utilization for simple algorithms on complex and varied hardware. CuDNN is enormous because it embeds precompiled binaries of many different compute kernels, times many variations of each kernel specialized for different data sizes and/or fused with other ker…

While I think you raise important points about the dominance of hardware optimizations, I think you're massively overstating the simplicity of the algorithms. Sure, it's easy to code the forward pass of a fully connected neural network, but writing code to train a useful modern architecture is a very different endeavor.

I disagree. I mean it's not trivial but it is completely within reach of a single person. The only part you'd really need to lean on libraries for would be data loading (e.g. jpeg). The core neural net stuff really is not that complex, even in the latest architectures like transformers or diffusion models. Look at stuff like George Hotz's tinygrad or Andrej Karpathy's makemore.

Re: PyTorch 2.0

#55

Post by brrrrrm »

claytonjy wrote: post #50
modeless wrote: post #44

Earlier quoted context omitted.

The complexity of deep learning algorithms is low but the complexity of the hardware is high. The problem solved by these gigabytes of libraries is getting peak utilization for simple algorithms on complex and varied hardware. CuDNN is enormous because it embeds precompiled binaries of many different compute kernels, times many variations of each kernel specialized for different data sizes and/or fused with other ker…

While I think you raise important points about the dominance of hardware optimizations, I think you're massively overstating the simplicity of the algorithms. Sure, it's easy to code the forward pass of a fully connected neural network, but writing code to train a useful modern architecture is a very different endeavor.

I disagree, the burden is almost exclusively maintaining fast implementations of primitive operators for all hardware. These ML libraries are collections of pure functions with minimal interfaces. There's very little code interdependence and it's not particularly difficult to implement modern algorithms to train networks.

full stable diffusion in https://github.com/geohot/tinygrad/blob/4fb97b8de0e210cc3778...

autograd in https://github.com/geohot/tinygrad/blob/4fb97b8de0e210cc3778...

Adam in https://github.com/geohot/tinygrad/blob/4fb97b8de0e210cc3778...

Re: PyTorch 2.0

#57

Post by wging »

hintymad wrote: post #25

A big lesson I learned from PyTorch vs other frameworks is that productivity trumps incremental performance improvement. Both Caffe and MXNet marketed themselves for being fast, yet apparently being faster here and here by some percentage simply didn't matter that much. On the other hand, once we make a system work and make it popular, the community will close the performance gap sooner than competitors expect. Anoth…

> Meta reportedly had more than 300 (?) people working on PyTorch,

How much did this change after the big Meta layoffs? I think I know people who are no longer there, but I haven't talked to them about it yet.

Re: PyTorch 2.0

#58

Post by windsignaling »

brap wrote: post #37
whimsicalism wrote: post #17

Earlier quoted context omitted.

I was reading this and thinking it was a pretty terrible answer - glad it is just generated by an AI and not you personally so I'm not insulting you. JAX is basically numpy on steroids and lets you do a lot of non-standard things (like a differentiable physics simulation or something) that would be harder with Pytorch. They are both "high-performance." Pytorch is more geared towards traditional deep learning and has…

I’m not sure why, but I realized it was AI from the very first sentence, not exaggerating. It’s just not something someone on HN would write.

Yup. Reminds me of an article you'd find in the top 10 Google search results...

Re: PyTorch 2.0

#59

Post by amelius »

modeless wrote: post #44
amelius wrote: post #41

One thing I'm noticing lately is that these DL libraries and their supporting libraries are getting unwieldy large, and difficult to version-manage. In my mind, DL is doing little more than performing some inner-products between tensors, so I'm curious why we should have libraries such as libcudnn, libcudart, libcublas, torch, etc. containing gigabytes of executable code. I just checked and I have 2.4GB (!!) of cuda-…

The complexity of deep learning algorithms is low but the complexity of the hardware is high. The problem solved by these gigabytes of libraries is getting peak utilization for simple algorithms on complex and varied hardware. CuDNN is enormous because it embeds precompiled binaries of many different compute kernels, times many variations of each kernel specialized for different data sizes and/or fused with other ker…

Ok, is there any way to trim down the amount of code used without reducing the performance of my particular application, and my particular machine?

I have the feeling that it's an all-or-nothing proposition. Either you have a simple CPU-only algorithm, or you have several gigabytes of libraries you don't really need.

Also, in some applications I would be willing to give up 10% of performance if I could reclaim 90% of space.

Re: PyTorch 2.0

#60

Post by modeless »

amelius wrote: post #59
modeless wrote: post #44

Earlier quoted context omitted.

The complexity of deep learning algorithms is low but the complexity of the hardware is high. The problem solved by these gigabytes of libraries is getting peak utilization for simple algorithms on complex and varied hardware. CuDNN is enormous because it embeds precompiled binaries of many different compute kernels, times many variations of each kernel specialized for different data sizes and/or fused with other ker…

Ok, is there any way to trim down the amount of code used without reducing the performance of my particular application, and my particular machine? I have the feeling that it's an all-or-nothing proposition. Either you have a simple CPU-only algorithm, or you have several gigabytes of libraries you don't really need. Also, in some applications I would be willing to give up 10% of performance if I could reclaim 90% of…

CuDNN is only for Nvidia GPUs, and those machines generally have decent sized disks and decent network connections so no nobody cares about a few GBs of libraries. There are alternatives to using CuDNN with much smaller binary size. Maybe they can match or beat it or maybe not, depending on your model and hardware. But you'll have to do your own work to switch to them, since most people are happy enough with CuDNN for now.

The real problem with deep learning on Nvidia is the Linux driver situation. Ugh. Hopefully one day they will come to their senses.

Post reply on HN