Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Tinygrad: A simple and powerful neural network framework

tinygrad.org

Post reply on HN
Print view View original

61–70 of 147 posts

Re: Tinygrad: A simple and powerful neural network framework

#61

Post by brrrrrm »

> It compiles a custom kernel for every operation, allowing extreme shape specialization.

This doesn't matter. Just look at the performance achieved by CuDNN kernels (which back PyTorch), they're dynamically shaped and hit near peak. For dense linear algebra at the size of modern neural networks, optimizing for the loop bound condition won't help much.

> All tensors are lazy, so it can aggressively fuse operations.

This matters. PyTorch teams are trying to implement that now (they have LazyTensor, AITemplate, TorchDynamo), but I'm not sure of the status (it's been tried repeatedly).

> The backend is 10x+ simpler, meaning optimizing one kernel makes everything fast.

The first part of that sentence matters, the second part doesn't. Kernels are already fast and their reuse outside of being fused into each other (which you need a full linear algebra compiler to do) isn't very high. If you make sum fast, you have not made matrix multiplication fast even though MM has a sum in it. It just isn't that easy to compose operations and still hit 80+% of hardware efficiency.

But it is easier to iterate fast and build a seamless lazy compiler if your backend is simple. You can pattern match more easily and ensure you handle edge cases without insanely complicated things like alias analysis (which PyTorch has to do).

Re: Tinygrad: A simple and powerful neural network framework

#62

Post by jerpint »

orlp wrote: post #29

> It's extremely simple, and breaks down the most complex networks into 4 OpTypes: > > - UnaryOps operate on one tensor and run elementwise. RELU, LOG, RECIPROCAL, etc... > - BinaryOps operate on two tensors and run elementwise to return one. ADD, MUL, etc... > - ReduceOps operate on one tensor and return a smaller tensor. SUM, MAX > - MovementOps operate on one tensor and move the data around, copy-free with ShapeTr…

Just looking at the code from my phone, but it seems that the conv op calls another primitive and einsum, which I believe is just a fancy MUL with broadcasting? so it might still be technically correct?

Re: Tinygrad: A simple and powerful neural network framework

#63

Post by terafo »

matesz wrote: post #9

If anybody is dealing with procrastination watch George Hotz live streaming 10h straight working on this library [1][2]. Does he take some supplements to do this? There is even 19.5h stream [3]. Actually I have local obs setup to record myself, just instead of streaming I do recordings for my own inspection. Important part is to do the inspection after. It works wonders. [1] https://youtu.be/GXy5eVwnL_Q [2] https://m…

It isn't 19.5 hour stream, I went and randomly clicked on couple of timestamps and stumbled upon[1]. So it's two almost-10-hour-long streams put together because they are thematically similar.

[1] https://youtu.be/xc0jGZYFQLQ?t=34333

Re: Tinygrad: A simple and powerful neural network framework

#64

Post by HelloNurse »

DeathArrow wrote: post #3

I believe neural networks are over hyped sometimes. They are not always the best tool for the job. There are lots of other ML techniques such as SVM, naive Bayes, k-nearest neighbor, decision tree, logistic regression, random forest etc. nobody is using because they lack the hype factor. If something lacks some keywords like neural network, deep learning, reinforced learning, than it is deemed not cool.

This is a library for neural networks, and it should be compared to other neural networks solutions.

Re: Tinygrad: A simple and powerful neural network framework

#65

Post by terafo »

langsoul-com wrote: post #34
matesz wrote: post #9

If anybody is dealing with procrastination watch George Hotz live streaming 10h straight working on this library [1][2]. Does he take some supplements to do this? There is even 19.5h stream [3]. Actually I have local obs setup to record myself, just instead of streaming I do recordings for my own inspection. Important part is to do the inspection after. It works wonders. [1] https://youtu.be/GXy5eVwnL_Q [2] https://m…

What's the file size for your recordings? 5 hours of 720p would be huge.

It wouldn't be huge, I once recorded a week of me using my pc(so around 80 hours in total), and it was sub-100 gigs. It was 1080p with decent quality, don't remember FPS though.

Re: Tinygrad: A simple and powerful neural network framework

#66

Post by nisegami »

nextlevelwizard wrote: post #26
pyinstallwoes wrote: post #17

Earlier quoted context omitted.

If it's not Adderall I don't know. But, if I've ever focused for that long it's been because of Ritalin or Adderall.

Is 10 hours really _that_ strange? You are (hopefully) focusing 8 hours "straight" during work _every day_. If you watch Hotz's streams he takes small breaks to talk with chat and to meme around (just like everyone else during their work days) and he eats lunch and whatever (again just like everyone else). What I'm trying to say is that Hotz's isn't a superman on Adderall he is just working on stuff he is excited abo…

>You are (hopefully) focusing 8 hours "straight" during work _every day_.

A single, continuous 30-minute stint of focus is probably a once-a-quarter event for me.

Re: Tinygrad: A simple and powerful neural network framework

#68

Post by Vetch »

nl wrote: post #57
Vetch wrote: post #56

Earlier quoted context omitted.

> NN based sentiment analysis is certainly a lot better than non-NN based techniques. I wouldn't say this. Sentiment analysis trained on the standard datasets is one place where performance is barely better than old-school linear classifiers. They remained brittle and easy to trick until recent flexible systems systems based on question answering, zero-shot entailment or lotsa instruction finetuning (improving in tha…

> Sentiment analysis trained on the standard datasets is one place where performance is barely better than old-school linear classifiers Well yeah. But why would you do that? Do what eveyrone does: Train on large scale a language corpus (or use a pre-trained model) then finetune for sentiment analysis. > I strongly advice against using something fine-tuned solely on sentiment datasets Did you mean trained on sentimen…

No, I meant finetuned. I also meant finetuned when I said trained. Experience with applying finetuned sentiment classifiers on real world data found gain vs cost of running to not be worth it. They remain nearly as brittle as cheaper classifiers and have a habit of gloming too much unto certain adjectives. They are also prone to overfitting on finetuned data's domain. Transformers trained not specifically on sentiment but on general domains like question answering or entailment are just leagues better for sentiment tasks.

Re: Tinygrad: A simple and powerful neural network framework

#69

Post by PartiallyTyped »

jerpint wrote: post #62
orlp wrote: post #29

> It's extremely simple, and breaks down the most complex networks into 4 OpTypes: > > - UnaryOps operate on one tensor and run elementwise. RELU, LOG, RECIPROCAL, etc... > - BinaryOps operate on two tensors and run elementwise to return one. ADD, MUL, etc... > - ReduceOps operate on one tensor and return a smaller tensor. SUM, MAX > - MovementOps operate on one tensor and move the data around, copy-free with ShapeTr…

Just looking at the code from my phone, but it seems that the conv op calls another primitive and einsum, which I believe is just a fancy MUL with broadcasting? so it might still be technically correct?

Einsum is an expressive way of doing element wise products and then possibly reducing them. An einsum is essentially a description of the dimensions of the input tensors and the dimensions of the resulting output after multiplication. If the output has reduced dimensions, then a summation is applied over them. The package einops provides reductions such as summation, averaging, and so on.

For example; the einsum " b k n p, k -> b k n p" broadcasts the second tensor b to b[None, :, None, None] and does element wise multiplication. It can be changed to a vector product by writing "b k n p, k -> b n p", which for all intents and purposes is identical to a.transpose(0, 2, 3, 1) @ b .

I can easily recommend the einops package and using einsum, simplifies things significantly.

Re: Tinygrad: A simple and powerful neural network framework

#70

Post by terafo »

stephc_int13 wrote: post #58

I understand that the Python code is mostly driving faster low-level code, but I wonder how much time is effectively wasted by not using a lower-level language. From my experience with game engines, it often turns out to be a bad idea (for performance and maintainability) to mix C/C++ and Lua or C#.

I would argue that there are performance *benefits* for a developer in running python code, due to how programs are run in python(Jupyter notebooks) you basically can change program on the fly, and not recompile and restart it, as you would do with compiled languages. And yeah, CPU does very very little in modern DL workloads and it is commonplace for CPU python code to be jitted and vectorized, so performance difference isn't as large as you would think.
Post reply on HN