Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

PyTorch 2.0

pytorch.org

Post reply on HN
Print view View original

31–40 of 111 posts

Re: PyTorch 2.0

#31

Post by synergy20 »

microtonal wrote: post #29
quietbritishjim wrote: post #5

> Today, we announce torch.compile, a feature that pushes PyTorch performance to new heights and starts the move for parts of PyTorch from C++ back into Python. I'll admit I don't know enough about PyTorch to know what torch.compile is exactly. But does this means some features of PyTorch will no longer be available in the core C++ library? One of the nice things about PyTorch had been that you could do your training…

One of the nice things about PyTorch had been that you could do your training in Python then deploy with a pure C++ application. Or even train in C++ or Rust without much loss in functionality.

Rust really has not had any presence in AI training engine yet, it's probably 100% c++.

Re: PyTorch 2.0

#32

Post by uoaei »

whimsicalism wrote: post #17
staunch wrote: post #14

Earlier quoted context omitted.

PyTorch and JAX are both open-source libraries for developing machine learning models, but they have some important differences. PyTorch is a more general-purpose library that provides a wide range of functionalities for developing and training machine learning models. It also has strong support for deep learning and is used by many researchers and companies in production environments. JAX, on the other hand, is desi…

I was reading this and thinking it was a pretty terrible answer - glad it is just generated by an AI and not you personally so I'm not insulting you. JAX is basically numpy on steroids and lets you do a lot of non-standard things (like a differentiable physics simulation or something) that would be harder with Pytorch. They are both "high-performance." Pytorch is more geared towards traditional deep learning and has…

Can someone comment more on what makes JAX that much better for differentiable simulations than PyTorch?

I'm working on a new module for work and none of my colleagues have much experience developing ML per se. I'm trying to decide whether to force their hand by implementing v1 in PyTorch or JAX and differentiable physics simulations is a likely future use case. Why is PyTorch harder?

Re: PyTorch 2.0

#33

Post by whimsicalism »

uoaei wrote: post #32
whimsicalism wrote: post #17

Earlier quoted context omitted.

I was reading this and thinking it was a pretty terrible answer - glad it is just generated by an AI and not you personally so I'm not insulting you. JAX is basically numpy on steroids and lets you do a lot of non-standard things (like a differentiable physics simulation or something) that would be harder with Pytorch. They are both "high-performance." Pytorch is more geared towards traditional deep learning and has…

Can someone comment more on what makes JAX that much better for differentiable simulations than PyTorch? I'm working on a new module for work and none of my colleagues have much experience developing ML per se. I'm trying to decide whether to force their hand by implementing v1 in PyTorch or JAX and differentiable physics simulations is a likely future use case. Why is PyTorch harder?

Because the `jax.numpy` operations & primitives are almost 1:1 with numpy, many working scientists who already have experience working with numpy will be able to figure out jax faster.

It is also easier to rewrite existing code/snippets (say you were working on a non-differentiable simulator before) into jax if you already have them in numpy then to do the whole rewrite in pytorch.

I will say that I think pytorch has improved its numpy compatability a lot in recent years, functions that I was convinced didn't exist with pytorch (like eigh) apparently actually do.

Re: PyTorch 2.0

#34

Post by dekhn »

whimsicalism wrote: post #27
dekhn wrote: post #26

Earlier quoted context omitted.

jax is not numpy on steroids. jax is "use python idiomatically to generate optimized XLA code for evaluating functions both forward and backward."

Probably the primary use of jax is `jax.numpy` which is XLA accelerated and differentiable numpy. I'll admit that saying "basically numpy on steroids" might have been an overreduction. It is a system for function transformations that is built on XLA and oriented towards science & ML applications. It's not just me saying stuff like this. François Chollet (creator of Keras): "[jax is] basically Numpy with gradients. An…

Yes- and that gradient part is a key detail that makes it more than "numpy on steroids". numpy on steroids would be a hardware accelerator that took numpy calls and made them return more quickly, but without the command-and-control and compile-python-to-xla aspects.

Re: PyTorch 2.0

#35

Post by whimsicalism »

dekhn wrote: post #34
whimsicalism wrote: post #27

Earlier quoted context omitted.

Probably the primary use of jax is `jax.numpy` which is XLA accelerated and differentiable numpy. I'll admit that saying "basically numpy on steroids" might have been an overreduction. It is a system for function transformations that is built on XLA and oriented towards science & ML applications. It's not just me saying stuff like this. François Chollet (creator of Keras): "[jax is] basically Numpy with gradients. An…

Yes- and that gradient part is a key detail that makes it more than "numpy on steroids". numpy on steroids would be a hardware accelerator that took numpy calls and made them return more quickly, but without the command-and-control and compile-python-to-xla aspects.

Well clearly I meant steroids of the gradient-developing variety.

I think you are being far too pedantic about what a biological compound would analogously do to a software library, especially given that I mention the differentiability property in the same sentence you are taking issue with.

Re: PyTorch 2.0

#36

Post by belval »

js2 wrote: post #28
belval wrote: post #7

> We believe that this is a substantial new direction for PyTorch – hence we call it 2.0. torch.compile is a fully additive (and optional) feature and hence 2.0 is 100% backward compatible by definition. How about just calling it PyTorch 1.14 if it's backward compatible? Version numbering shouldn't be used as a marketing gimmick.

Dismissive comments like this make me not want to read HN anymore and in addition it’s against the HN guidelines: It’s snarky. It’s incurious. It’s neither thoughtful nor substantive. It’s flame bait. It’s a shallow dismissal. It doesn’t teach anything. It’s the most provocative thing to complain about. https://news.ycombinator.com/newsguidelines.html I’m sorry I had to leave this comment, so let me also try to respo…

Personal attack aside, from your own link:

> Given a version number MAJOR.MINOR.PATCH, increment the:

> MAJOR version when you make incompatible API changes

> MINOR version when you add functionality in a backwards compatible manner

> PATCH version when you make backwards compatible bug fixes

> Additional labels for pre-release and build metadata are available as extensions to the MAJOR.MINOR.PATCH format.

You can point towards some other details, but it doesn't change the fact that for the overwhelming majority of people, the quote above is what semver is. Besides, my original comment does not say "They broke semver", it says they shouldn't bump the major version if they don't make backward incompatible change because afterwards the mental model of "Can I use version X.Y.Z?" is broken.

When TensorFlow moved to 2.0 it's because they were changing from graphs and session definition to eager mode. That makes sense, that means the underlying API and how the downstream users interact with it changed. These are just newer features that, while very useful, have limited bearing on downstream users.

Re: PyTorch 2.0

#37

Post by brap »

whimsicalism wrote: post #17
staunch wrote: post #14

Earlier quoted context omitted.

PyTorch and JAX are both open-source libraries for developing machine learning models, but they have some important differences. PyTorch is a more general-purpose library that provides a wide range of functionalities for developing and training machine learning models. It also has strong support for deep learning and is used by many researchers and companies in production environments. JAX, on the other hand, is desi…

I was reading this and thinking it was a pretty terrible answer - glad it is just generated by an AI and not you personally so I'm not insulting you. JAX is basically numpy on steroids and lets you do a lot of non-standard things (like a differentiable physics simulation or something) that would be harder with Pytorch. They are both "high-performance." Pytorch is more geared towards traditional deep learning and has…

I’m not sure why, but I realized it was AI from the very first sentence, not exaggerating. It’s just not something someone on HN would write.

Re: PyTorch 2.0

#38

Post by dekhn »

whimsicalism wrote: post #35
dekhn wrote: post #34

Earlier quoted context omitted.

Yes- and that gradient part is a key detail that makes it more than "numpy on steroids". numpy on steroids would be a hardware accelerator that took numpy calls and made them return more quickly, but without the command-and-control and compile-python-to-xla aspects.

Well clearly I meant steroids of the gradient-developing variety. I think you are being far too pedantic about what a biological compound would analogously do to a software library, especially given that I mention the differentiability property in the same sentence you are taking issue with.

OK, actually as long as it's gradient-developing steroids, I'll allow it.

Re: PyTorch 2.0

#39

Post by ansk »

So this looks like a further convergence of the tensorflow and pytorch APIs (the lower-level APIs at least). Tensorflow was designed with compilable graphs as the primary execution model and as part of their 2.0 release, they redesigned the APIs to encompass eager execution as well. Pytorch is coming from the other end, with eager execution being the default and now emphasizing improved tools for graph compilation in their 2.0 release. The key differentiator going forward seems to be that tensorflow is using XLA as their compiler and pytorch is developing their own toolset for compilation. As someone who cares far more about performance than API ergonomics, the quality of the compiler is the main selling point for me and I'll gladly switch over to whatever framework is winning in the compiler race. Does anyone know of any benchmarks comparing the performance of pytorch's compilers with XLA?

Re: PyTorch 2.0

#40

Post by amelius »

ansk wrote: post #39

So this looks like a further convergence of the tensorflow and pytorch APIs (the lower-level APIs at least). Tensorflow was designed with compilable graphs as the primary execution model and as part of their 2.0 release, they redesigned the APIs to encompass eager execution as well. Pytorch is coming from the other end, with eager execution being the default and now emphasizing improved tools for graph compilation in…

How much performance can be squeezed from going from the plain python API to the graph-based solution, typically?
Post reply on HN