Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

What I wish someone had told me about tensor computation libraries

eigenfoo.xyz

Post reply on HN
Print view View original

41–50 of 89 posts

Re: What I wish someone had told me about tensor computation libraries

#41

Post by alevskaya »

hyperbovine wrote: post #13
komuher wrote: post #8

Earlier quoted context omitted.

Not even cloese, jax.jit allow you to compute almost anything using lax.for_loops, lax.cond and other lax and jax contsturts pytorch jit does not allow that its just extra optimization for static pytorch functions.

No autodiff for most of these though.

JAX autograd will work on most any jitted fn - the control-flow limitations are no autograd for code with for/while loops since there's a statically unknowable trip count through the loop body. Much looping code can be handled differentiably using a "scan" though.

Re: What I wish someone had told me about tensor computation libraries

#42

Post by timkpaine »

prideout wrote: post #34

Are these libraries ever useful in non-deep learning applications? It sounds like Theano is a bit more general purpose, but why would I ever need it outside of a deep learning context? I wonder if it could be used for something crazy, e.g. setting up a graph that generates shadertoy-like images on the GPU.

Idk about using these libraries, but its almost impossible to find generic graph libraries that aren't designed around either ML or alternatively scheduling batches. One such example is my own, https://github.com/timkpaine/tributary

Re: What I wish someone had told me about tensor computation libraries

#43

Post by alevskaya »

jstrong wrote: post #40

I'm a theano diehard, and I'll never get over how google came along, introduced a shittier version of theano, garnered worldwide acclaim for it, and killed the better library in the process.

Having written and debugged both Theano and TF plenty in the past, I think this is a somewhat uncharitable take, esp. recalling the absolutely enormous Theano compile times. :) I think Theano was genius, but a system that relied on python-string-based C++ code-emitters was always going to have trouble with long-term sustainability.

Re: What I wish someone had told me about tensor computation libraries

#45

Post by blt »

albertzeyer wrote: post #19

I was not aware that the PyMC developers have forked and continued Theano: https://github.com/pymc-devs/Theano-PyMC It seems very active right now. Here some further information: https://pymc-devs.medium.com/the-future-of-pymc3-or-theano-i... I haven't really found references to its new name "Aesara". Apparently, the main new feature for Theano will be the JAX backend. I wonder though, my experience when working with…

Re. the last point, was trying to think of computations where 1) an efficient in-place version is possible, and 2) the most efficient out-of-place version is significantly faster than copying the input and executing the in-place version.

In 1D convolutions, the in-place version would need to use O(filter size) scratch space for lookahead, but this doesn't seem like it would be too significant. However, it might start to become significant in higher-dimensional convolutions.

Any particular example that occurs to you?

Re: What I wish someone had told me about tensor computation libraries

#46

Post by webmaven »

magicfractal wrote: post #27
akhilcacharya wrote: post #18

Earlier quoted context omitted.

I’m truly baffled as to why such a sophisticated and useful package is being distributed and advertised by an anonymous individual.

Probably they’re afraid because it might be related to their day job :/

> Probably they’re afraid because it might be related to their day job :/

A slightly more common scenario is an employer that insists on "we own everything, related to your job or not, that you do even on your own time and equipment" clauses in employee contracts even though such clauses don't happen to be enforceable in the relevant jurisdiction.

Rather than having to "clear through your manager and legal" every little thing to get it added to your contract's personal IP whitelist, publishing anonymously makes perfect sense, where the plan is to de-anonymize after employment ends, at which point (should said now-former-employer have a hissy fit), their own counsel will eventually inform them they don't have a leg to stand on. After sending at least one threatening letter, of course.

Another solution is to spam your manager (and legal) with every trivial 'invention' that pops into your head until they relent[0][1], but that can burn though political capital you may prefer to use for other purposes, and will probably only narrow the scope rather than remove the unenforceable clause.

[0] https://cr.yp.to/patents/tarzian.html (my favorite is invention #12)

[1] As examples I was seriously tempted to use: "Python, but with 1-based indexing", "LinkExchange, but for Wingmen", and "ROT-13 Markdown".

Re: What I wish someone had told me about tensor computation libraries

#47

Post by josh2600 »

dragandj wrote: post #38
godelski wrote: post #32

Earlier quoted context omitted.

Thanks! That's helpful to know. I have no experience with Clojure/Lisp but a fair amount with C/C++ (minor in CUDA).

Please read a few of tutorials from my blog. Most programmers in your situation told me that they had no problems following it; it only gradually introduces advanced Clojure concepts, and code snippets are usually extremely short + completely executable interactively as-is.

This is a great site. A few pieces of unsolicited feedback from a marketing perspective:

* Always have the call to action repeated at the bottom of the page; you did a great job having it above the fold, but I got to the end and had to scroll back up to express intent. That's a flowbreaking design.

* This is purely stylistic, but I strongly prefer the first letter of each line to be capitalized. I find the stylistic inconsistency around capitalization offputting.

* You have pretty nicely made docs, consider making the link to view the sample chapters a clickable picture of the first diagram-page in your chapter.

Thanks for making this!

Re: What I wish someone had told me about tensor computation libraries

#48

Post by physicsyogi »

prideout wrote: post #34

Are these libraries ever useful in non-deep learning applications? It sounds like Theano is a bit more general purpose, but why would I ever need it outside of a deep learning context? I wonder if it could be used for something crazy, e.g. setting up a graph that generates shadertoy-like images on the GPU.

Libraries like this enable differentiable programming, which lets you backprop through more than just neural networks. For instance, people have built a differentiable raytracer and plugged a physics engine into reinforcement learning to accelerate training.

https://en.wikipedia.org/wiki/Differentiable_programming

Re: What I wish someone had told me about tensor computation libraries

#49

Post by wokwokwok »

hansvm wrote: post #39
wokwokwok wrote: post #29

Earlier quoted context omitted.

Concise isn’t always better. You’re throwing alway all the names of the arguments and using arbitrary words like “conv” to represent operations. This is typical bad clojure in my experience; write once, forget wtf the magic was, throw away and rewrite it again later. Clojure doesn’t have to be incomprehensible arcane magic that does everything in 10 lines. The more complex the code, the more important it is that what…

Concision is a style choice to be used with care. Spending screen space on additional characters and descriptions detracts from the ability to fit more logic on the screen at once and grok the larger flow. Splashing symbolic alphabet soup into your IDE in the name of concision isn't usually a good idea, but naming something "conv" in the immediate local context of a convolutional layer doesn't seem so bad.

Does 'convo' refer to a 2D convolution or a 1D convolution? Given the large number [1] of arguments that a convolution can take, which ones are being specified? I can probably guess, since only 2 are given, but if there were more, which order would they be in and which would refer to which?

The code is on github [2, 3] see for yourself if you think it's more or less obvious than the python equivalent of a 'trivial' network.

I would say 'conv2d' is probably reasonably standard in meaning; I refer to 'convo' as arbitrary, because it is. Either (ideally) avoid abbreviations, or use standard ones.

[1] - https://pytorch.org/docs/stable/generated/torch.nn.Conv2d.ht...

[1] - https://github.com/uncomplicate/deep-diamond/blob/master/tes...

[2] - https://github.com/uncomplicate/deep-diamond/blob/master/tes...

Re: What I wish someone had told me about tensor computation libraries

#50

Post by bravura »

alevskaya wrote: post #43
jstrong wrote: post #40

I'm a theano diehard, and I'll never get over how google came along, introduced a shittier version of theano, garnered worldwide acclaim for it, and killed the better library in the process.

Having written and debugged both Theano and TF plenty in the past, I think this is a somewhat uncharitable take, esp. recalling the absolutely enormous Theano compile times. :) I think Theano was genius, but a system that relied on python-string-based C++ code-emitters was always going to have trouble with long-term sustainability.

I am one of the authors of the Theano work. I am happy to hear that the Theano project is now being maintained again.

I will agree with alevskaya that the compilation times were an issue in my particular research ten years ago. I was trying to build neural-networks for parsing that were created at run-time. Since each parse tree had a different computation graph, I was not able to use Theano since it required compiling every single type of parse tree computation graph it encountered during training.

[edit if you want more details: There is really interesting old-school work called "Recursive distributed representations" and later "Labelling recursive auto-associative memory" that used auto-encoders to consume a variable length sequence, e.g. text string, in a sequential fashion. My work with Yoshua Bengio---incomplete---was based upon the idea of doing unsupervised binary parsing of sentences using a hierarchical RAAM-style approach: At any given point in time, greedily find the two adjacent tokens that could be most easily compressed into one token with low reconstruction error. However, once you apply this recursively and end up with auto-encoding binary parse trees, you end up with a variety of different computation graphs, each of which required separate compilation.]

Post reply on HN