Related: The Simple Essence of Automatic Differentiation - Conal Elliot
What I wish someone had told me about tensor computation libraries
61–70 of 89 posts
Re: What I wish someone had told me about tensor computation libraries
#62Earlier quoted context omitted.
Thanks! That's helpful to know. I have no experience with Clojure/Lisp but a fair amount with C/C++ (minor in CUDA).
Please read a few of tutorials from my blog. Most programmers in your situation told me that they had no problems following it; it only gradually introduces advanced Clojure concepts, and code snippets are usually extremely short + completely executable interactively as-is.
Thanks! I have this slated in my to-do tabs (which I actually do lol). Just looking through it looks like you did a lot of work and took a lot of time for this so just wanted to say thanks.
Re: What I wish someone had told me about tensor computation libraries
#63I was not aware that the PyMC developers have forked and continued Theano: https://github.com/pymc-devs/Theano-PyMC It seems very active right now. Here some further information: https://pymc-devs.medium.com/the-future-of-pymc3-or-theano-i... I haven't really found references to its new name "Aesara". Apparently, the main new feature for Theano will be the JAX backend. I wonder though, my experience when working with…
Hello, I'm the person spearheading this Theano fork! Your comments match my experience with the old Theano very well, so I have to respond.
> Apparently, the main new feature for Theano will be the JAX backend.
The JAX transpilation feature arose as a quick example of how flexible Theano can be, both in terms of its "hackability" and its simple yet effective foundation (i.e. "static" graphs). It's definitely not the main focus of the fork, but it is easily the newest feature that stands out at the user-level.
The points you raised about the old Theano are actually the main focus, and we've already made large internal changes that address a few of them directly. At the very least, nearly all of them are on the roadmap toward our new library named "Aesara".
The `Scan` `Op` and its optimizations are definitely going to change, and I have no intention of sacrificing improvements for backward compatibility, or anything else that would constrain the extent of improvements. I too have dealt with the difficulties involved in writing Scan optimizations (e.g. https://github.com/pymc-devs/symbolic-pymc/blob/master/symbo...) and am painfully aware of how unnecessary most of them are.
> - The graph building and esp the graph optimizations are very slow. This is because all the logic is done in pure Python. ...
The most important graph optimization performance problems are not actually related to Python performance; they're demonstrably design and implementation induced. That is unless you're talking exclusively about graphs so large they reach the "natural" limits of Python performance by definition. Even then, a nearly one-to-one C translation isn't likely to solve those scaling problems.
For example, the graph optimization/rewriting framework would require entire graphs to be copied at multiple points in the process, and this was almost completely due to some design oddities. We've already made all of the large-scale changes needed in order to remedy this design constraint, so we're well on our way to fixing that. See https://github.com/pymc-devs/Theano-PyMC/pull/158
The rewriting process also doesn't track or use node information very well (or at all), so the whole optimization process itself can take an unnecessary number of passes through a graph. For instance, its "local" optimizations have a "tracking" option that specifies the `Op` types to which they apply; however, that feature isn't even used unless the local optimizations are applied by a `LocalOptGroup`. I've noticed at least a few instances in which these local optimizations are applied to inapplicable `Op`s on each visit to a node. Worse yet, within `LocalOptGroup` those local optimizations aren't applied directly to the relevant `Op`s, even though the requisite `Op` type-to-node information is readily available. In other words, optimizations could be directly applied to the relevant nodes in these cases and dramatically reduce the amount of blind graph traversals performed.
At best, a reimplementation in a language with a better compiler, like C, would largely amount to a questionable brute-force attempt at performance, and the ease of manipulating graphs and developing graph rewrites would suffer. With Aesara, we're going for the opposite. We want a smarter framework and _more_ focus on domain-specific optimizations (e.g. linear/tensor algebra, statistics, computer science) from the domain experts themselves, so code transparency and ease of development really matters. When we need raw performance in specific areas of the code, we'll pinpoint those areas and write C extensions, in standard Python fashion.
> ... When switching to TensorFlow, building the graph felt almost instant in comparison. ...
Last I checked, TensorFlow had almost no default graph optimizations, aside from some basic CSE and minor canonicalization and algebraic simplifications in the `grappler` module, so it absolutely should be instantaneous. More importantly, TensorFlow isn't designed for graph rewriting, and definitely not at the Python level where rapid prototyping and testing is possible outside of Google.
Otherwise, if you're talking about initially _building_ a graph and not calling `theano.function`, there are no optimizations involved. Latency in that case would be something entirely different and well worth reproducing for an issue. If what you were observing was the effect of calling `theano.function`, the latency was most likely due to the C transpilation and subsequent compilation. That's a feature that necessarily takes time, but produces code that's often faster than TensorFlow even today.
In summary, the changes we're most focused on right now are for developers like yourself who have had to deal with the core of Theano, so, please, stop by the fork and help us make a better `Scan`!
Re: What I wish someone had told me about tensor computation libraries
#64Are these libraries ever useful in non-deep learning applications? It sounds like Theano is a bit more general purpose, but why would I ever need it outside of a deep learning context? I wonder if it could be used for something crazy, e.g. setting up a graph that generates shadertoy-like images on the GPU.
Idk about using these libraries, but its almost impossible to find generic graph libraries that aren't designed around either ML or alternatively scheduling batches. One such example is my own, https://github.com/timkpaine/tributary
Interesting library & idea, almost like its own programming paradigm when you abstract away all the specificity for building software or running ETL jobs or whatever.
But this is a completely different kind of graph. The graphs being discussed here are differentiable DAGs of mathematical computations.
Re: What I wish someone had told me about tensor computation libraries
#65Why are they called TENSOR computation libraries?
Re: What I wish someone had told me about tensor computation libraries
#66Earlier quoted context omitted.
Concision is a style choice to be used with care. Spending screen space on additional characters and descriptions detracts from the ability to fit more logic on the screen at once and grok the larger flow. Splashing symbolic alphabet soup into your IDE in the name of concision isn't usually a good idea, but naming something "conv" in the immediate local context of a convolutional layer doesn't seem so bad.
Does 'convo' refer to a 2D convolution or a 1D convolution? Given the large number [1] of arguments that a convolution can take, which ones are being specified? I can probably guess, since only 2 are given, but if there were more, which order would they be in and which would refer to which? The code is on github [2, 3] see for yourself if you think it's more or less obvious than the python equivalent of a 'trivial' n…
I know nothing about neural networks, but it looks to me like convo is smart enough to create the convolution of the correct dimension based on the arguments. Where as Keras seem to force you to use a different constructor for different dimensions. That explains why it's called convo, since it creates convolutions of any dimension.
Also, most options are provided as a map to convo as well it looks like, so you'd have similar named arguments for convo once you get to defining optional things like padding and strides.
Re: What I wish someone had told me about tensor computation libraries
#67Seems to have missed the existence of jax.jit, which basically constructs an XLA program (call it a graph if you like) from your Python function which can then be optimized.
In the section title, JAX: > But JAX even lets you just-in-time compile your own Python functions into XLA-optimized kernels...
The authors gives that quote (from the JAX documentation) but does not seem to interiorize it as his conclusion says:
> This is the niche that Theano (or rather, Theano-PyMC/Aesara) fills that other contemporary tensor computation libraries do not: the promise is that if you take the time to specify your computation up front and all at once, Theano can optimize the living daylight out of your computation - whether by graph manipulation, efficient compilation or something else entirely - and that this is something you would only need to do once.
It is exactly what JAX does. There is a computational graph in JAX (its encoded in XLA and specified with their numpy like syntax), it is build once, optimized and then runs on the GPU.
Re: What I wish someone had told me about tensor computation libraries
#68Earlier quoted context omitted.
If only C++ supported interactive REPL and the rest of Clojure/Lisp goodies, that might be possible. However, the code is CLOSELY related to the actual CUDA/C++ api. It's a lot simpler, concise, and everything, but I explain everything so that you can use the relevant parts with cuDNN and DNNL APIs in any language that you're most proficient in.
Does Cling C++ interpreter do what you want? https://github.com/root-project/cling
Have you used it? When I last tried Cling not that long ago, it wasn't even alpha quality software and given that it has been around for a while, my default assumption would be that this hasn't suddenly improved.
Re: What I wish someone had told me about tensor computation libraries
#69Let me chip in with some self-promotion. This book explains and executes every single line of code interactively, from low level operations to high-level networks that do everything automatically. The code is built on the state of the art performance operations of oneDNN (Intel, CPU) and cuDNN (CUDA, GPU). Very concise readable and understandable by humans. https://aiprobook.com/deep-learning-for-programmers/ Here's…
"Deep Learning in Clojure with Fewer Parentheses than Keras and Python" Love it! :D What better way to define a neural network in code than an S-expression?
I am not sure if I am that enthusiastic. The problem with Lisp is not a number of parenthesis but where they are and what is their role. In c-like languages parenthesis help parser compiler but they also help humans to read the code. In case of Lisp they are just for the sake of the parser.
Let's look on the code:
Python:
model = Sequential()
model.add(Conv2D(32, kernel_size=(3, 3),
activation='relu',
input_shape=(28, 28, 1)))
(defonce net-bp
(network (desc [128 1 28 28] :float :nchw)
Re: What I wish someone had told me about tensor computation libraries
#70Earlier quoted context omitted.
"Deep Learning in Clojure with Fewer Parentheses than Keras and Python" Love it! :D What better way to define a neural network in code than an S-expression?
I am not sure if I am that enthusiastic. The problem with Lisp is not a number of parenthesis but where they are and what is their role. In c-like languages parenthesis help parser compiler but they also help humans to read the code. In case of Lisp they are just for the sake of the parser. Let's look on the code: Python: model = Sequential() model.add(Conv2D(32, kernel_size=(3, 3), activation='relu', input_shape=(28…
> Which one is more readable?
Both are equally readable to me.
Now, granted 128 1 28 28 can be difficult to understand without documentation but that is not due to Lisp's fully parenthesized prefix notation. The Clojure code would also look equally readable if it had used keyword arguments.
Are you sure you are not confounding familiarity with readability? With Lisp, after a while, the parentheses become invisible to the programmer.