The last arguments about why you would want a static graph and even it's drawbacks and complaints sound basically similar to why you would want to do functional programming
What I wish someone had told me about tensor computation libraries
51–60 of 89 posts
Re: What I wish someone had told me about tensor computation libraries
#52I will say that I am very excited by the tftorch.py effort from @sillysaurusx: https://twitter.com/theshawwn/status/1311925180126511104
The idea being that pytorch can just be a high-level API executing lower-level tensorflow under the hood.
Re: What I wish someone had told me about tensor computation libraries
#53Tensorflow 1.0 has its roots in how Theano was built. Same thing, a statically built graph that is run through a compilation step, with a numpy-like API. So what makes Theano such an ingenious concept while TF is regarded as “programming through a keyhole”?
Here's my take about TF (in general, not particularly 1.x or 2.x): Like many things from Google, I always had the impression that the library, while better than alternatives at the time, is too tailored to Google use cases. And if you fall outside of them, bad luck. Still, at work we find it easier to deploy and interoperate with other tools than Pytorch. Hell, we have a guy working in Pytorch who converts his work t…
I recently used TF 1.0 (former Theano author, current PyTorch user) and found TF 1.0 to be hellaciously difficult to grok and seemed to include a lot of unnecessary abstractions.
There was existing TF 1.0 code I was trying to extract gradients through (nsynth-wavenet). I spent over 8 hours on it unsuccessfully; I asked for help from a friend at Google who worked on TF and he couldn't figure it out either. I emailed the original author of the code and he acknowledged that he didn't know how to do it either, and he had an old notebook he could dig up that kinda would work with a lot of fixes.
Re: What I wish someone had told me about tensor computation libraries
#54Earlier quoted context omitted.
Having written and debugged both Theano and TF plenty in the past, I think this is a somewhat uncharitable take, esp. recalling the absolutely enormous Theano compile times. :) I think Theano was genius, but a system that relied on python-string-based C++ code-emitters was always going to have trouble with long-term sustainability.
I am one of the authors of the Theano work. I am happy to hear that the Theano project is now being maintained again. I will agree with alevskaya that the compilation times were an issue in my particular research ten years ago. I was trying to build neural-networks for parsing that were created at run-time. Since each parse tree had a different computation graph, I was not able to use Theano since it required compili…
[deleted]
Re: What I wish someone had told me about tensor computation libraries
#55Earlier quoted context omitted.
Here's my take about TF (in general, not particularly 1.x or 2.x): Like many things from Google, I always had the impression that the library, while better than alternatives at the time, is too tailored to Google use cases. And if you fall outside of them, bad luck. Still, at work we find it easier to deploy and interoperate with other tools than Pytorch. Hell, we have a guy working in Pytorch who converts his work t…
I recently used TF 1.0 (former Theano author, current PyTorch user) and found TF 1.0 to be hellaciously difficult to grok and seemed to include a lot of unnecessary abstractions. There was existing TF 1.0 code I was trying to extract gradients through (nsynth-wavenet). I spent over 8 hours on it unsuccessfully; I asked for help from a friend at Google who worked on TF and he couldn't figure it out either. I emailed t…
Also see my comment here: https://news.ycombinator.com/item?id=25439073
I am definitely interested in a higher-level Pytorch API that uses TF as an execution engine.
Re: What I wish someone had told me about tensor computation libraries
#56Earlier quoted context omitted.
The goal of NN-512 is efficient neural net inference on inexpensive, CPU-only cloud compute instances For example, a Skylake-X cloud compute instance costs $10 per CPU-core per month at Vultr, and the NN-512 generated code does about 18 DenseNet121 inferences per CPU-core per second (in series, not batched) In contrast, GPU cloud compute is almost unbelievably expensive. Even Linode charges $1000 per month, or $1.50…
I'm not disagreeing with you. I acknowledge that there may be a market for CPU-only NN tasks. I think a thorough benchmark, either by you or by someone else, will only help your case, by giving a clear picture to those who need to make a decision. Fun fact, GPUs are massively under-utilized during NN training. So it's quite possible NN on a good CPU might be only slightly slower.
GPU underutilization depends on what, exactly, the model you're training is. It's not unreasonable to hit 80% or more of CUDA core usage on non-recurrent models like convnets, given sufficiently fast data pipelines and a reasonable batch size. Transformers and other recurrent functions hit 100% CUDA core utilization for large portions of each epoch, with the low-% usage on the comparatively short weight update at the end. As well, the current rule of thumb is that at the same price point (so a Xeon 4114 and a Nvidia Titan RTX) the GPU completes each epoch in 10% of the time as the CPU given the same compute graph... So it's highly unlikely that training will be anywhere close to as fast on a CPU as it is on a GPU.
Re: What I wish someone had told me about tensor computation libraries
#57NN-512 ( https://NN-512.com ) Generate fully vectorized, stand-alone, human-readable C99 code for neural net inference, and understand exactly what's happening. For example, watch the code run with Linux's perf top and see the relative costs of each layer of the computation. Total transparency, no dependencies outside the C POSIX library
I just want to say that I'm very interested in this library and have commented on it before. I'd really like to see it reach feature parity with pytorch or theano and emit your C++ code on the backend.
For example, I am not aware that one can currently use your library to implement Wavenet, other audio generative models like Wavegrad, or transformers.
Keep up the good work.
Re: What I wish someone had told me about tensor computation libraries
#58[deleted]
Re: What I wish someone had told me about tensor computation libraries
#59Let me chip in with some self-promotion. This book explains and executes every single line of code interactively, from low level operations to high-level networks that do everything automatically. The code is built on the state of the art performance operations of oneDNN (Intel, CPU) and cuDNN (CUDA, GPU). Very concise readable and understandable by humans. https://aiprobook.com/deep-learning-for-programmers/ Here's…
Machine Learning in Clojure reminds me of Yann LeCun’s ML course from 2010, where we used an adorable language called Lush:
which I suppose can best be described as Lisp and Python having a baby. It was immense fun to code neural networks from scratch in it. I hope Clojure can find a bigger place in the world of ML.
Re: What I wish someone had told me about tensor computation libraries
#60Earlier quoted context omitted.
I’m truly baffled as to why such a sophisticated and useful package is being distributed and advertised by an anonymous individual.
Probably they’re afraid because it might be related to their day job :/
it need not be related to your job. some employer might ask that since you're skilled enough to do such thing, then you should have been performing extraordinarily on the job, even if you are already delivering what the job asks for, and just as good as your peers. at worst, some struggling poorly managed startup might even "turnaround" and eventually you don't own your side passion project anymore.