Related: http://tensor-compiler.org/codegen.html. This converts an expression in tensor index notation into executable code.
Tensor Comprehensions
41–50 of 64 posts
Re: Tensor Comprehensions
#42Earlier quoted context omitted.
Yes, this is just jargon specific to the HPC / numerical analysis field.
And not without meaning. HPC people hold themselves to higher standards than the average Joes throwing stuff at the currently-fashionable "continuous integration server" until the lights turn green. "Codes" are often things that have been proven, written on a legal pad, typed into the machine, and finally validated on a set of test problems. Could you prove that your program has second-order numerical stability? http…
A shining example of the Dunning–Kruger effect. Scientific code is some of the worst-engineered code you'll ever deal with because it's written by people that are great at physics but terrible at software engineering.
It will be poorly documented, have single letter var names everywhere without comments, have a bunch of hidden dependencies on whatever the postdoc had installed on his/her laptop, leak memory but 'terminate before it should be a problem', etc. Oh, and there will almost never be even a single regression test.
Sure, the underlying algorithm might have a nice proof of numerical stability, but the actual software is almost always trash until some poor soul has to re-engineer it if the software turns out to be useful.
Re: Tensor Comprehensions
#43Could someone please explain how this compares to the TensorFlow approach? I can only assume that it's omitted from the article due to marketing reasons.
My understanding is that Tensor Comprehensions provides a way to automatically generate optimized CUDA code for algorithms written in a high-level language that more closely mirrors the notation used in mathematical formulas. So you could use it to automatically find more optimal low-level implementations for components used in libraries such as PyTorch and TensorFlow which usually call out to hand-written low-level…
How can it be both general, and fast? For example, CuDNN ops are fast because they are very specialized and highly tuned. Is convolution written with TC going to be as fast as CuDNN convolution?
Or, if TC's strength is in its generality, then what are the advantages over something like CuPy for Chainer?
Can someone give an example where TC shines?
Re: Tensor Comprehensions
#44I'm a fan of evolutionary algorithms, but are they really effective enough here to be comparable to an engineer tuning code? They might be able to find a good configuration of a few canned options but real optimization often requires some creativity or at least an understanding of the hardware. Will certainly be interesting to see this in practice!
The crucial part is the polyhedral optimizer which does indeed include several GPU-specific heuristics (multilevel parallelization, coalescing, etc) and specialization to tensor sizes. Evolutionary autotuner is used to tweak the parameters of the optimizer. As a result, TC can beat cublas and cudnn on certain networks; details in the report.
What would be a relationship between TC and something like CuPy?
Re: Tensor Comprehensions
#45Earlier quoted context omitted.
My understanding is that Tensor Comprehensions provides a way to automatically generate optimized CUDA code for algorithms written in a high-level language that more closely mirrors the notation used in mathematical formulas. So you could use it to automatically find more optimal low-level implementations for components used in libraries such as PyTorch and TensorFlow which usually call out to hand-written low-level…
How can it be both general, and fast? For example, CuDNN ops are fast because they are very specialized and highly tuned. Is convolution written with TC going to be as fast as CuDNN convolution? Or, if TC's strength is in its generality, then what are the advantages over something like CuPy for Chainer? Can someone give an example where TC shines?
With cuDNN, some expert had to sit down and write an optimized kernel for your ops. With TC, the idea is you write the simple, tensor comprehension for your op, and then throw the autotuner at it, and get something that approaches or even beats the hand-tuned example. The diagrams in the paper give a sense for some of these situations.
Re: Tensor Comprehensions
#46Facebook "research" --> this is sooooo cute!
Re: Tensor Comprehensions
#47Earlier quoted context omitted.
The crucial part is the polyhedral optimizer which does indeed include several GPU-specific heuristics (multilevel parallelization, coalescing, etc) and specialization to tensor sizes. Evolutionary autotuner is used to tweak the parameters of the optimizer. As a result, TC can beat cublas and cudnn on certain networks; details in the report.
What would be a relationship between TC and something like CuPy?
CuPy itself is just a framework, and you could slot TC in as a thing that generates operators for it. CuPy also famously has support for inline CUDA kernels; the equivalent TC kernels are shorter and autotunable.
Re: Tensor Comprehensions
#48Earlier quoted context omitted.
My understanding is that Tensor Comprehensions provides a way to automatically generate optimized CUDA code for algorithms written in a high-level language that more closely mirrors the notation used in mathematical formulas. So you could use it to automatically find more optimal low-level implementations for components used in libraries such as PyTorch and TensorFlow which usually call out to hand-written low-level…
How can it be both general, and fast? For example, CuDNN ops are fast because they are very specialized and highly tuned. Is convolution written with TC going to be as fast as CuDNN convolution? Or, if TC's strength is in its generality, then what are the advantages over something like CuPy for Chainer? Can someone give an example where TC shines?
Section 7 of the paper (https://arxiv.org/abs/1802.04730) has a couple of examples.
In short, yes CuDNN is fast for the cases it was tuned for. It is probably faster on power-of-two sizes, but when you operate on a 26 x 1024954 x 3 tensor, TC can generate specialized code. Want 42 x 17 x 5? TC can generate differently specialized code. With almost no effort from the user (or performance engineers).
Can a performance expert do better job than TC optimizer? Very likely yes, but it will very likely take much more time.
TC is not a framework. It can be integrated with any framework of your liking.
Re: Tensor Comprehensions
#49Related: http://tensor-compiler.org/codegen.html . This converts an expression in tensor index notation into executable code.
Sure, it is one of the works we cite. It seems to be mostly targeted at sparse computations and does not have GPU support.
Tensor Comprehensions does not try to manage memory and thus can be integrated into DL frameworks easily.
Re: Tensor Comprehensions
#50Earlier quoted context omitted.
code:codes::math:maths?
In the Commonwealth we say maths but we still say code. I've noticed we also tend to also write "computer program" the US way, despite writing "TV programme".
Yup, British-educated here: programme for sequences of activities, program for the thing the computer runs... it’s just a little oddity I’ve noticed I’ve picked up and seem to be quite consistently applying.