Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

PyTorch – Tensors and Dynamic neural networks in Python

pytorch.org

Post reply on HN
Print view View original

51–60 of 91 posts

Re: PyTorch – Tensors and Dynamic neural networks in Python

#51

Post by divbit »

smhx wrote: post #46
tsomctl wrote: post #41

Earlier quoted context omitted.

Not only that, but it appears to use the same core c libray (TH) as Lua torch.

we actually share the same git-subtree between Lua and Python variants. TH, THNN, THC, THCUNN are shared.

I have been running in the back of my mind the idea of attempting a julialang interface to torch for a few weeks now, using the ccall interface: http://docs.julialang.org/en/release-0.5/manual/calling-c-an.... Do you have any thoughts / recommendations w'r't' that? (This would be more of a fun / weekend(s) project for me than anything else) My goal would be to have the tensors override the .* and * operators as used here: https://gist.github.com/divbit/ec57ad2f1989bf13aecdf9e1e1056...

Re: PyTorch – Tensors and Dynamic neural networks in Python

#53

Post by nickdavidhaynes »

gallerdude wrote: post #52

What's the highest level neural network lib I can use? I'm a total programming idiot but I find neural nets fascinating.

http://playground.tensorflow.org/

Pretty much no way to use neural networks (except for playing, like above) without writing code.

Re: PyTorch – Tensors and Dynamic neural networks in Python

#54

Post by Smerity »

PieSquared wrote: post #48
Smerity wrote: post #33

Only a few months ago people saying that the deep learning library ecosystem was starting to stabilize. I never saw that as the case. The latest frontier for deep learning libraries is ensuring efficient support for dynamic computation graphs. Dynamic computation graphs arise whenever the amount of work that needs to be done is variable. This may be when we're processing text, one example being a few words while anot…

Could you elaborate on what you find lacking in TensorFlow? I regularly use TensorFlow for exactly these sorts of dynamic graphs, and it seems to work fairly well; I haven't used Chainer or DyNet extensively, so I'm curious to see what I'm missing!

When you say "exactly these sorts of dynamic graphs", what do you mean? TensorFlow has support for dynamic length RNN unrolling but that really doesn't extend well to any dynamic graph structure such as recursive tree structure creation. Since the computation graph has a different shape and size for every input they are difficult to batch and any pre-defined static graph is likely excessive, wasting computation, or inexpressive.

The primary issue is that the computation graph is not imperative - you define it explicitly. Chainer describes this as the difference between "Define-and-Run" frameworks and "Define-by-Run" frameworks[1].

TensorFlow is "Define-and-Run". For loops and conditionals end up needing to be defined and injected into the graph structure before it's run. This means there are "tf.while_loop" operations for example - you can't use a "while" loop as it exists in Python or C++. This makes debugging difficult as the process of defining the computation graph is separate to the usage of it and also restricts the flexibility of the model.

In comparison, both Chainer, PyTorch, and DyNet are "Define-by-Run", meaning the graph structure is defined on-the-fly via the actual forward computation. This is a far more natural style of programming. If you perform a for loop in Python, you're actually performing a for loop in the graph structure as well.

This has been a large enough issue that, very recently, a team at Google created "TensorFlow Fold"[2], still unreleased and unpublished, that handles dynamic computation graphs. In it they tackle specifically dynamic batching within the tree structured LSTM architecture.

If you compare the best example of recursive neural networks in TensorFlow[3] (quite complex and finicky in the details) to the example that comes with Chainer[4], which is perfectly Pythonic and standard code, it's pretty clear why one might prefer "Define-by-Run" ;)

[1]: http://docs.chainer.org/en/stable/tutorial/basic.html

[2]: https://openreview.net/pdf?id=ryrGawqex

[3]: https://github.com/bogatyy/cs224d/tree/master/assignment3

[4]: https://github.com/pfnet/chainer/blob/master/examples/sentim...

Re: PyTorch – Tensors and Dynamic neural networks in Python

#55

Post by superfx »

Smerity wrote: post #33

Only a few months ago people saying that the deep learning library ecosystem was starting to stabilize. I never saw that as the case. The latest frontier for deep learning libraries is ensuring efficient support for dynamic computation graphs. Dynamic computation graphs arise whenever the amount of work that needs to be done is variable. This may be when we're processing text, one example being a few words while anot…

How is adding dynamic graphs to TensorFlow "after the fact" while adding it to Torch isn't? (Torch is much older than TF).

Re: PyTorch – Tensors and Dynamic neural networks in Python

#56

Post by infinite8s »

vegabook wrote: post #38
cr0sh wrote: post #35

Earlier quoted context omitted.

> And lua is miles faster than Python when you're outside the tensor domain, ie while you're sourcing and wrangling your data. Then use Lua for that, if you are more comfortable there and want/need the speed bump. There's nothing that says an entire project or whatnot has to be developed in a singular language. Use each tool to its strengths, as your needs, requirements, and abilities dictate.

There always has to be someone rolling out the horses-for-courses pitch. No. I wanted Lua to gain traction with other people . That's the point. I would have liked the Lua sci-ecosystem to be healthy as an alternative.

Is there an equivalent to numpy in the Lua space?

Re: PyTorch – Tensors and Dynamic neural networks in Python

#57

Post by Smerity »

superfx wrote: post #55
Smerity wrote: post #33

Only a few months ago people saying that the deep learning library ecosystem was starting to stabilize. I never saw that as the case. The latest frontier for deep learning libraries is ensuring efficient support for dynamic computation graphs. Dynamic computation graphs arise whenever the amount of work that needs to be done is variable. This may be when we're processing text, one example being a few words while anot…

How is adding dynamic graphs to TensorFlow "after the fact" while adding it to Torch isn't? (Torch is much older than TF).

Torch was never written as a static graph computation framework. Torch was/is more a tensor manipulation library where you are executing the individual operations step by step and the graph can be tracked and constructed incrementally from those operations. For this reason, much of PyTorch is about building a layer on top of the underlying components (which are focused on efficiently manipulating tensors and/or implementing low level NN components) rather than re-architecting Torch itself.

This won't be the same for TensorFlow as it was written with the concept of a static computation graph at its core. I'm certainly not saying it's impossible to re-architect - and many smart people in the community and at Google are devoting thinking and code to it - but simply that the process will be far more painful as it was not written with this as an intended purpose.

To note - there are many advantages to static computation graphs. Of particular interest to Google is that they distribute their computations very effectively over large amounts of hardware. Being able to do this with a dynamic computation graph would be far more problematic.

Re: PyTorch – Tensors and Dynamic neural networks in Python

#60

Post by baq »

vegabook wrote: post #34
jjawssd wrote: post #31

Earlier quoted context omitted.

A very large portion of performance problems can be mitigated with the use of cython and the new asyncio stuff. asyncio success story: https://magic.io/blog/asyncpg-1m-rows-from-postgres-to-pytho... cython: http://scikit-learn.org/stable/developers/performance.html

Luajit is at least 10x faster than python and easily obviates the need to mess around with cython. That's an easy win for Lua. Let's be honest: Torch has decided that if you cannot beat them, join them. It is about network effects. Not about Python better than Lua intrinsically.

but why do you care if luajit is faster than python if everything that matters is computed on the GPU anyways?
Post reply on HN