Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

What I wish someone had told me about tensor computation libraries

eigenfoo.xyz

Post reply on HN
Print view View original

81–89 of 89 posts

Re: What I wish someone had told me about tensor computation libraries

#81

Post by runetech »

dragandj wrote: post #14

Let me chip in with some self-promotion. This book explains and executes every single line of code interactively, from low level operations to high-level networks that do everything automatically. The code is built on the state of the art performance operations of oneDNN (Intel, CPU) and cuDNN (CUDA, GPU). Very concise readable and understandable by humans. https://aiprobook.com/deep-learning-for-programmers/ Here's…

I really enjoy your work. I bought your book recently and appreciate your approach of building an understanding of the library based on "first principles". Really appreciate this performant and elegant option for working with deep-learning in Clojure - Thank you!

Re: What I wish someone had told me about tensor computation libraries

#82

Post by dragandj »

runetech wrote: post #81
dragandj wrote: post #14

Let me chip in with some self-promotion. This book explains and executes every single line of code interactively, from low level operations to high-level networks that do everything automatically. The code is built on the state of the art performance operations of oneDNN (Intel, CPU) and cuDNN (CUDA, GPU). Very concise readable and understandable by humans. https://aiprobook.com/deep-learning-for-programmers/ Here's…

I really enjoy your work. I bought your book recently and appreciate your approach of building an understanding of the library based on "first principles". Really appreciate this performant and elegant option for working with deep-learning in Clojure - Thank you!

Thanks!

Re: What I wish someone had told me about tensor computation libraries

#83

Post by dr_zoidberg »

bravura wrote: post #55
bravura wrote: post #53

Earlier quoted context omitted.

I recently used TF 1.0 (former Theano author, current PyTorch user) and found TF 1.0 to be hellaciously difficult to grok and seemed to include a lot of unnecessary abstractions. There was existing TF 1.0 code I was trying to extract gradients through (nsynth-wavenet). I spent over 8 hours on it unsuccessfully; I asked for help from a friend at Google who worked on TF and he couldn't figure it out either. I emailed t…

Also see my comment here: https://news.ycombinator.com/item?id=25439073 I am definitely interested in a higher-level Pytorch API that uses TF as an execution engine.

My coworker said that he basically started from this article[0] and then adapted a few things to his workflow. He also said that learnopencv "covers like 70% of what you really have to do and you have to figure the rest out, not hard but may take you some time".

[0] https://www.learnopencv.com/pytorch-to-tensorflow-model-conv...

Re: What I wish someone had told me about tensor computation libraries

#84

Post by PoignardAzur »

Can someone ELI5 what are the differences between the different libraries are? The article uses a lot of jargon, an something that frustrates me about getting into machine learning is that teaching material will either abstract away what the internals do or assume that you already know how the internals work.

Some specific questions:

> They provide ways of specifying and building computational graphs

Is the article talking about neural networks? As in, arrays of arrays of weights, where input values go through successive layers, and for each layer the same instruction is applied to some values with the respective weight?

Or is it talking about a graph as in, a functional graph, where manually written functions call other manually written functions? (hence why a later paragraph talks about if-else statements and for loops)

> Almost all tensor computation libraries support autodifferentiation in some capacity (either forward-mode, backward-mode, or both).

What are those?

From the wikipedia article, it sounds like autodifferentiation basically means running f(x+dx)-f(x), but if there are entire frameworks handling it, then there's probably something fancier going on.

> According to the JAX quickstart, JAX bills itself as “NumPy on the CPU, GPU, and TPU, with great automatic differentiation for high-performance machine learning research”. Hence, its focus is heavily on autodifferentiation.

The earlier description makes it sound like JAX does some cutting-edge compilation stuff to transform semi-arbitrary functions (with ifs and else and loops and stuff) into a function that returns it derivative.

So how can that stuff run on the GPU? It sounds like there would be a lot of branching code.

And how is that related to machine learning / neural networks?

Re: What I wish someone had told me about tensor computation libraries

#85

Post by dekhn »

patrec wrote: post #68
dekhn wrote: post #25

Earlier quoted context omitted.

Does Cling C++ interpreter do what you want? https://github.com/root-project/cling

Have you used it? When I last tried Cling not that long ago, it wasn't even alpha quality software and given that it has been around for a while, my default assumption would be that this hasn't suddenly improved.

it's used by ROOT at CERN, so it's unlikely it's "alpha quality software" since it's being used for production science at scale.

Re: What I wish someone had told me about tensor computation libraries

#86

Post by timkpaine »

nerdponx wrote: post #64
timkpaine wrote: post #42

Earlier quoted context omitted.

Idk about using these libraries, but its almost impossible to find generic graph libraries that aren't designed around either ML or alternatively scheduling batches. One such example is my own, https://github.com/timkpaine/tributary

Interesting library & idea, almost like its own programming paradigm when you abstract away all the specificity for building software or running ETL jobs or whatever. But this is a completely different kind of graph. The graphs being discussed here are differentiable DAGs of mathematical computations.

Is it that different? https://github.com/timkpaine/tributary/blob/main/docs/exampl...

Re: What I wish someone had told me about tensor computation libraries

#87

Post by lr1970 »

drzoltar wrote: post #59
dragandj wrote: post #14

Let me chip in with some self-promotion. This book explains and executes every single line of code interactively, from low level operations to high-level networks that do everything automatically. The code is built on the state of the art performance operations of oneDNN (Intel, CPU) and cuDNN (CUDA, GPU). Very concise readable and understandable by humans. https://aiprobook.com/deep-learning-for-programmers/ Here's…

Machine Learning in Clojure reminds me of Yann LeCun’s ML course from 2010, where we used an adorable language called Lush: http://lush.sourceforge.net/ which I suppose can best be described as Lisp and Python having a baby. It was immense fun to code neural networks from scratch in it. I hope Clojure can find a bigger place in the world of ML.

In those days there was a lovely LuaJIT based tensor manipulation language torch7 [1,2] developed by Leon Bottou. It later became basis for PyTorch. I still believe that Lua in general and LuaJIT in particular are much superior to Python for Deep Learning.

[1] http://torch.ch/

[2] https://github.com/torch/torch7

Re: What I wish someone had told me about tensor computation libraries

#88

Post by nerdponx »

timkpaine wrote: post #86
nerdponx wrote: post #64

Earlier quoted context omitted.

Interesting library & idea, almost like its own programming paradigm when you abstract away all the specificity for building software or running ETL jobs or whatever. But this is a completely different kind of graph. The graphs being discussed here are differentiable DAGs of mathematical computations.

Is it that different? https://github.com/timkpaine/tributary/blob/main/docs/exampl...

Now that is definitely interesting. And you have some notion of "differentiability" for all of your various sources, sinks, and transforms?

That said, Tensorflow and Pytorch are both very much general purpose numerical computing libraries. You don't have to use them for neural networks.

Post reply on HN