Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

PyTorch 2.0

pytorch.org

Post reply on HN
Print view View original

81–90 of 111 posts

Re: PyTorch 2.0

#81

Post by brrrrrm »

ansk wrote: post #39

So this looks like a further convergence of the tensorflow and pytorch APIs (the lower-level APIs at least). Tensorflow was designed with compilable graphs as the primary execution model and as part of their 2.0 release, they redesigned the APIs to encompass eager execution as well. Pytorch is coming from the other end, with eager execution being the default and now emphasizing improved tools for graph compilation in…

Some context/history:

For compiler people reading this, a lot of common compiler terms have been entirely reinvented in the context of machine learning frameworks. An ML "graph" refers almost exactly to the dataflow graph (DFG) of a program. TensorFlow 1.0 only exposed a DFG, which is well known to be far simpler to apply optimizations to (assuming you have a linear algebra compiler).

PyTorch integrated with Python (an interpreted language) and does not expose an underlying DFG. This is labeled "eager" and means that compilation of PyTorch requires optimization over both the control flow graph (CFG) and DFG. Python by default exposes neither of these things in a standard way. Some ML workloads simplify easily to a DFG (torch FX can handle this), but the general case does not. Although TorchScript (a subset of Python) tackled the CFG in 1.0, the team is now taking it further and compiling Python byte-code itself (with torchdynamo), which means you don't need to change any code and still get compilation speed ups! That's why 2.0 is significant.

Of course, all of this requires a linear algebra compiler to actually do the optimizations which is why things like AITemplate (for inference) and TorchInductor (which calls into a bunch of other compilers for training) exist for PyTorch. TensorFlow's linear algebra compiler is XLA.

Re: PyTorch 2.0

#82

Post by qayxc »

anigbrowl wrote: post #79

I get that Nvidia is the favorite GPU (because CUDA) and that library maintainers want to chase the latest and best to do the most. But I don't get why support for older hardware (including CUDA stuff) is just deprecated or abandoned, nor why support for other GPU architectures is lacking across many popular ML libraries. A lot of this is down to driver availability and software stacks...but is all of it? Game engine…

NVIDIA invested billions in their software infrastructure over the past 15 years (CUDA was first released in 2007) and basically brute forced their way into academia back in 2008, sponsoring labs around the world to get GPGPU going.

Both mindshare and market share (outside of super computers) are just overwhelming at this point. Their market share in the consumer market is ~86% as of this quarter. The data centre market is quite fragmented when it comes to accelerators, but in AI training, NVIDIA is still the market leader.

> Game engines/engineers seem to be able to be productive on a wide variety of GPU hardware, why do so many ML libraries just not provide any support at all?

That's a different kettle of fish. Game engines rely on the graphics driver's implementations of low- and mid level APIs like Vulkan or Direct3D.

The brunt of the work is also often performed by the middle-ware (mostly Unreal Engine and Unity or in-house engines like Frostbite) that had been in development for decades; with most games focusing on high level optimisation wrt. the middle-ware used.

ML-frameworks on the other hand need to optimise compute kernels as well as data flow between host CPU and accelerator (e.g. GPU). This involves hand-tuning algorithms to best match specific GPU architectures, while most shaders used in games are basically the same across all GPU vendors and it's the vendors themselves who do the fine tuning and per-game optimisations in their graphics drivers (hence the obscene sizes of GPU drivers these days).

While that's a good enough approach for games, it's simply not possible to do the same for ML-models. There's just too much flexibility (no API that dictates which calls do what, when, and how) to make general ML-optimisations at the driver level.

> How come every ML library doesn't at least have an OpenCL fallback or the like?

OpenCL is horrible to work with and stopped being properly supported by vendors (e.g. newer versions are rarely being implemented and optimised). The difference between CUDA and OpenCL from an implementor's perspective is that CUDA works seamlessly with surrounding C++ code and compute kernels can be embedded in the host CPU code base. OpenCL on the other hand is modelled after the ancient OpenGL 2.x paradigm and requires tedious setup and careful integration (checking capabilities and all that jazz). OpenCL is basically dead at this point.

There are alternatives to CUDA, but most frameworks rely heavily on the highly optimised libraries NVIDIA ships (e.g. CuDNN) and don't have the resources to implement the functionality themselves. Some hardware vendors offer proprietary backends, like Apple or Intel and you just have to wait for them to catch up. AMD has ROCm, but that's more of a drop-in replacement that aims at running CUDA code on AMD cards.

Re: PyTorch 2.0

#83

Post by roenxi »

vikinghckr wrote: post #70
hintymad wrote: post #25

A big lesson I learned from PyTorch vs other frameworks is that productivity trumps incremental performance improvement. Both Caffe and MXNet marketed themselves for being fast, yet apparently being faster here and here by some percentage simply didn't matter that much. On the other hand, once we make a system work and make it popular, the community will close the performance gap sooner than competitors expect. Anoth…

What I really find interesting here is that PyTorch, a library maintained by Facebook, is winning the marketshare and mindshare due to clean API, whereas Tensorflow, maintained by Google, is losing due to inferior API. In general, Google as a company emphasizes code quality and best practices far more than Facebook. But the story was reversed here.

Google's stereotype is the company that can handle and manage complexity - things like Kubernetes, indexing the internet, etc. They are clumsy at persuading people to use their products and have a patchy history of launching platforms that people want to use. Google+ and Google cloud vs AWS spring to mind, Kubernetes is a good platform but challenging to learn. Chrome is an unusual aberration where they did a great job. Android is maybe also a strong counterexample, not sure what the state of play was there.

But whatever their code quality may be, luring in devs to their API has seen hits and misses.

Re: PyTorch 2.0

#84

Post by ShamelessC »

vikinghckr wrote: post #70
hintymad wrote: post #25

A big lesson I learned from PyTorch vs other frameworks is that productivity trumps incremental performance improvement. Both Caffe and MXNet marketed themselves for being fast, yet apparently being faster here and here by some percentage simply didn't matter that much. On the other hand, once we make a system work and make it popular, the community will close the performance gap sooner than competitors expect. Anoth…

What I really find interesting here is that PyTorch, a library maintained by Facebook, is winning the marketshare and mindshare due to clean API, whereas Tensorflow, maintained by Google, is losing due to inferior API. In general, Google as a company emphasizes code quality and best practices far more than Facebook. But the story was reversed here.

Google historically has a culture of writing libraries with other Google employees as the target audience.

Re: PyTorch 2.0

#85

Post by anigbrowl »

qayxc wrote: post #82
anigbrowl wrote: post #79

I get that Nvidia is the favorite GPU (because CUDA) and that library maintainers want to chase the latest and best to do the most. But I don't get why support for older hardware (including CUDA stuff) is just deprecated or abandoned, nor why support for other GPU architectures is lacking across many popular ML libraries. A lot of this is down to driver availability and software stacks...but is all of it? Game engine…

NVIDIA invested billions in their software infrastructure over the past 15 years (CUDA was first released in 2007) and basically brute forced their way into academia back in 2008, sponsoring labs around the world to get GPGPU going. Both mindshare and market share (outside of super computers) are just overwhelming at this point. Their market share in the consumer market is ~86% as of this quarter. The data centre mar…

Thanks much for this in-depth explanation. I've been struggling with this for a while as I am low on the learning curve with a lot of ML stuff (partly due to headaches with finding a GPU that was affordable and properly supported in order to develop elementary competence). I think I'm just going to get an eGPU with a recent & decent CUDA card rather than waiting for a utopia of interoperability and backwards compatibility.

Re: PyTorch 2.0

#86

Post by antixk »

la_fayette wrote: post #53
SleekEagle wrote: post #45

Earlier quoted context omitted.

Exactly, especially in the age of ridiculously rapid development that we have found ourselves in over the past few years. This is exactly why TensorFlow is dying

:) it still is impossible to bring a cnn rnn network in pytorch to mobile, which works fine with tflite...

Not sure if you have heard of Pytorch Mobile but it is very possible[0]

[0] https://pytorch.org/tutorials/beginner/deeplabv3_on_ios.html

Re: PyTorch 2.0

#87

Post by ngcc_hk »

Meta is sort of strange. They bet a lot of unusual language and invest a lot on others. And unlike google which just try this that (which also have a good point as we need innovation, just do not assume it will exist a few month/year down the road). Sad the strange taste of that thing. At least it is not as disruptive like Twitter and only hurt itself.

Re: PyTorch 2.0

#88

Post by new_user55 »

chazeon wrote: post #43
uoaei wrote: post #32

Earlier quoted context omitted.

Can someone comment more on what makes JAX that much better for differentiable simulations than PyTorch? I'm working on a new module for work and none of my colleagues have much experience developing ML per se. I'm trying to decide whether to force their hand by implementing v1 in PyTorch or JAX and differentiable physics simulations is a likely future use case. Why is PyTorch harder?

I have seen JAX-MD[1] but not sure about "much better". On the other hand, there is just no MD implemented with PyTorch. [1]: https://github.com/jax-md/jax-md

There is [torch-md](https://github.com/torchmd/torchmd)

Re: PyTorch 2.0

#90

Post by aschleck »

vikinghckr wrote: post #70
hintymad wrote: post #25

A big lesson I learned from PyTorch vs other frameworks is that productivity trumps incremental performance improvement. Both Caffe and MXNet marketed themselves for being fast, yet apparently being faster here and here by some percentage simply didn't matter that much. On the other hand, once we make a system work and make it popular, the community will close the performance gap sooner than competitors expect. Anoth…

What I really find interesting here is that PyTorch, a library maintained by Facebook, is winning the marketshare and mindshare due to clean API, whereas Tensorflow, maintained by Google, is losing due to inferior API. In general, Google as a company emphasizes code quality and best practices far more than Facebook. But the story was reversed here.

It's worth noting that Google is also working on JAX these days and that's picking up steam, and one reason is because its API is super clean.
Post reply on HN