Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Tensor Comprehensions

research.fb.com

Post reply on HN
Print view View original

11–20 of 64 posts

Re: Tensor Comprehensions

#11

Post by ngvrnd »

grandmczeb wrote: post #4
wrs wrote: post #2

>produce the high-performance codes that the machine learning community needs Somewhat OT, but I've been wondering for a long time… Is the HPC community the only place the word "codes" is used like this? In usual CS parlance programming is done using a substance called "code" ("the high-performance code the community needs"), but in HPC literature the word "codes" is used, as if programming consisted of distinct obje…

Code is generally considered a mass noun among software engineers, but "codes" is pretty commonly used by academics, especially in other disciplines. In particular, physicists and mathematicians seem to use it pretty frequently, so that might explain why some in the HPC community use it as well. I've also noticed that it seems more common among Europeans, but that might be just personal experience.

Also as in "simulation codes". I associate it with physicists and Fortran programs, rightly or wrongly.

Re: Tensor Comprehensions

#12

Post by nicovasilache »

jabl wrote: post #9

Slightly resembles the Tensor Contraction Engine for quantum chemistry/physics ( http://www.csc.lsu.edu/%7Egb/TCE/ ). Although it predates this by a couple of decades.

Hello, Tensor Comprehensions absolutely use techniques that have been existing for a few years (Halide) or many decades (polyhedral model, Einstein notation(century old?), ...). The TCE is definitely also a motivational prior work which also uses the polyhedral model for optimizing loop nests. What we tried to achieve here, is a solid research tool to make a subset of underlying optimizations algorithms usable in practice by non-experts. One such optimization algorithm is described in our joint 2011 PoPL paper with authors of TCE (Loop transformations: convexity, pruning and optimization).

Re: Tensor Comprehensions

#13

Post by ozinenko »

tehsauce wrote: post #7

I'm a fan of evolutionary algorithms, but are they really effective enough here to be comparable to an engineer tuning code? They might be able to find a good configuration of a few canned options but real optimization often requires some creativity or at least an understanding of the hardware. Will certainly be interesting to see this in practice!

The crucial part is the polyhedral optimizer which does indeed include several GPU-specific heuristics (multilevel parallelization, coalescing, etc) and specialization to tensor sizes. Evolutionary autotuner is used to tweak the parameters of the optimizer. As a result, TC can beat cublas and cudnn on certain networks; details in the report.

Re: Tensor Comprehensions

#14

Post by alexbeloi »

The most surprising thing to me is that they can parameterize a nontrivial section of the implementation space of a function, or that such a section exists that hasn't been optimized away by the compiler.

Re: Tensor Comprehensions

#15

Post by nicovasilache »

tehsauce wrote: post #7

I'm a fan of evolutionary algorithms, but are they really effective enough here to be comparable to an engineer tuning code? They might be able to find a good configuration of a few canned options but real optimization often requires some creativity or at least an understanding of the hardware. Will certainly be interesting to see this in practice!

Hello, evolutionary algorithms by themselves I am not sure and don't have enough experience with atm. In our particular context the key is the intermediate representation on which the evolution happens and the compiler behaviors it triggers. We are still far from competitive against highly tuned expert code in computation-bound regimes but we do see >50% shared memory BW peak usage in nvprof in multiple cases. This first iteration is aimed at addressing the immediate research productivity needs: no need to write hundreds of lines of framework integration and low-level kernels to get a reasonable CUDA performance. We think we are helping remove the first immediate bottleneck that are very frustrating in practice to ML researchers when their layer does not translate exactly in supported, optimized, vendor libraries.

Re: Tensor Comprehensions

#16

Post by r00fus »

grandmczeb wrote: post #4
wrs wrote: post #2

>produce the high-performance codes that the machine learning community needs Somewhat OT, but I've been wondering for a long time… Is the HPC community the only place the word "codes" is used like this? In usual CS parlance programming is done using a substance called "code" ("the high-performance code the community needs"), but in HPC literature the word "codes" is used, as if programming consisted of distinct obje…

Code is generally considered a mass noun among software engineers, but "codes" is pretty commonly used by academics, especially in other disciplines. In particular, physicists and mathematicians seem to use it pretty frequently, so that might explain why some in the HPC community use it as well. I've also noticed that it seems more common among Europeans, but that might be just personal experience.

code:codes::math:maths?

Re: Tensor Comprehensions

#17

Post by fdej »

wrs wrote: post #2

>produce the high-performance codes that the machine learning community needs Somewhat OT, but I've been wondering for a long time… Is the HPC community the only place the word "codes" is used like this? In usual CS parlance programming is done using a substance called "code" ("the high-performance code the community needs"), but in HPC literature the word "codes" is used, as if programming consisted of distinct obje…

Yes, this is just jargon specific to the HPC / numerical analysis field.

Re: Tensor Comprehensions

#18

Post by DoofusOfDeath »

wrs wrote: post #2

>produce the high-performance codes that the machine learning community needs Somewhat OT, but I've been wondering for a long time… Is the HPC community the only place the word "codes" is used like this? In usual CS parlance programming is done using a substance called "code" ("the high-performance code the community needs"), but in HPC literature the word "codes" is used, as if programming consisted of distinct obje…

In computational acoustics, I've seen "codes" used more commonly, especially by those who started out in the punchcard days.

Re: Tensor Comprehensions

#19

Post by nicovasilache »

DoofusOfDeath wrote: post #18
wrs wrote: post #2

>produce the high-performance codes that the machine learning community needs Somewhat OT, but I've been wondering for a long time… Is the HPC community the only place the word "codes" is used like this? In usual CS parlance programming is done using a substance called "code" ("the high-performance code the community needs"), but in HPC literature the word "codes" is used, as if programming consisted of distinct obje…

In computational acoustics, I've seen "codes" used more commonly, especially by those who started out in the punchcard days.

punchcards .. days .. ouch that's painful :)

Re: Tensor Comprehensions

#20

Post by nicovasilache »

alexbeloi wrote: post #14

The most surprising thing to me is that they can parameterize a nontrivial section of the implementation space of a function, or that such a section exists that hasn't been optimized away by the compiler.

The thing is that compilers usually quickly go to SSA form and it is not the best IR to optimize loops in. Then you fight it to extract a high-level IR and this little process makes it very easy to lose high-level information. We don't do this so we begin from a friendlier starting point.
Post reply on HN