Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Tinygrad: A simple and powerful neural network framework

tinygrad.org

Post reply on HN
Print view View original

51–60 of 147 posts

Re: Tinygrad: A simple and powerful neural network framework

#51

Post by liuliu »

orlp wrote: post #29

> It's extremely simple, and breaks down the most complex networks into 4 OpTypes: > > - UnaryOps operate on one tensor and run elementwise. RELU, LOG, RECIPROCAL, etc... > - BinaryOps operate on two tensors and run elementwise to return one. ADD, MUL, etc... > - ReduceOps operate on one tensor and return a smaller tensor. SUM, MAX > - MovementOps operate on one tensor and move the data around, copy-free with ShapeTr…

Very similar idea as Jittor, convolution definitely can be break down: https://github.com/Jittor/jittor/blob/master/python/jittor/n...

Re: Tinygrad: A simple and powerful neural network framework

#52

Post by marcyb5st »

jstx1 wrote: post #30
learndeeply wrote: post #6

Earlier quoted context omitted.

I can't think of anything that neural nets can't beat, except small tabular data with boosted decision trees. Can you give some examples?

(I don't really agree with GP's point but for the sake of answering your question) 1. Collaborative filtering based on a sparse dataset of implicit interactions. 2. Many time series applications.

Didn't all recommendations engines move to two-towers like models? I remember that it "solved" the freshness problem (ie when adding a new item to your catalog how do you recommend it to users if there are no ratings/interactions). Of course as long as you have a good model that creates items embeddings.

Regarding time series, don't everyone moved to attention based models?

Not challenging your answer, just curious. I work mostly with Graph NNs and quite a bit out of touch with the rest of the field.

Re: Tinygrad: A simple and powerful neural network framework

#53

Post by anonymoushn »

nextlevelwizard wrote: post #26
pyinstallwoes wrote: post #17

Earlier quoted context omitted.

If it's not Adderall I don't know. But, if I've ever focused for that long it's been because of Ritalin or Adderall.

Is 10 hours really _that_ strange? You are (hopefully) focusing 8 hours "straight" during work _every day_. If you watch Hotz's streams he takes small breaks to talk with chat and to meme around (just like everyone else during their work days) and he eats lunch and whatever (again just like everyone else). What I'm trying to say is that Hotz's isn't a superman on Adderall he is just working on stuff he is excited abo…

At most workplaces you are interrupted dozens of times per day and have big time blocks of stuff that prevents focus. Where do you work?

Re: Tinygrad: A simple and powerful neural network framework

#55

Post by alexmolas »

ordu wrote: post #42
alexmolas wrote: post #37

> almost 9000 GitHub stars I wouldn't say that 7500 stars is almost 9000 stars ;)

It is probably based on a meme https://en.wikipedia.org/wiki/It%27s_Over_9000 ! They are not over 9k yet but closing.

ah, I didn't get the reference haha

anyway, I just gave them my star ;)

Re: Tinygrad: A simple and powerful neural network framework

#56

Post by Vetch »

nl wrote: post #50
DeathArrow wrote: post #11

Earlier quoted context omitted.

>I'm no expert but can you show how those techniques can be used to solve the same problems NNs can? Sentiment analysis, classification.

NN based sentiment analysis is certainly a lot better than non-NN based techniques. Classification depends on the problem (and mostly the datasize). Boosting is certainly competitive on tabular data and widely everywhere I've worked. No one talks about it (except on Kaggle) because it's pretty much at a local maximum. All the improvement comes from manual feature engineering. But modern techniques using NNs on tabula…

> NN based sentiment analysis is certainly a lot better than non-NN based techniques.

I wouldn't say this. Sentiment analysis trained on the standard datasets is one place where performance is barely better than old-school linear classifiers. They remained brittle and easy to trick until recent flexible systems systems based on question answering, zero-shot entailment or lotsa instruction finetuning (improving in that order). I strongly advice against using something fine-tuned solely on sentiment datasets. It'd be a total waste.

Re: Tinygrad: A simple and powerful neural network framework

#57

Post by nl »

Vetch wrote: post #56
nl wrote: post #50

Earlier quoted context omitted.

NN based sentiment analysis is certainly a lot better than non-NN based techniques. Classification depends on the problem (and mostly the datasize). Boosting is certainly competitive on tabular data and widely everywhere I've worked. No one talks about it (except on Kaggle) because it's pretty much at a local maximum. All the improvement comes from manual feature engineering. But modern techniques using NNs on tabula…

> NN based sentiment analysis is certainly a lot better than non-NN based techniques. I wouldn't say this. Sentiment analysis trained on the standard datasets is one place where performance is barely better than old-school linear classifiers. They remained brittle and easy to trick until recent flexible systems systems based on question answering, zero-shot entailment or lotsa instruction finetuning (improving in tha…

> Sentiment analysis trained on the standard datasets is one place where performance is barely better than old-school linear classifiers

Well yeah. But why would you do that?

Do what eveyrone does: Train on large scale a language corpus (or use a pre-trained model) then finetune for sentiment analysis.

> I strongly advice against using something fine-tuned solely on sentiment datasets

Did you mean trained on sentiment datasets? I agree with that.

Otherwise, well [1] is a decent overview of the field. I think Document Vectors using Cosine Similarity[2] at 17 is the highest rated that isn't a NN trained on large corpus and fine-tune on sentiment task. Even that uses document vectors that are trained on a large language corpus.

[1] https://paperswithcode.com/sota/sentiment-analysis-on-imdb

[2] https://paperswithcode.com/paper/the-document-vectors-using-...

Re: Tinygrad: A simple and powerful neural network framework

#58

Post by stephc_int13 »

I understand that the Python code is mostly driving faster low-level code, but I wonder how much time is effectively wasted by not using a lower-level language.

From my experience with game engines, it often turns out to be a bad idea (for performance and maintainability) to mix C/C++ and Lua or C#.

Re: Tinygrad: A simple and powerful neural network framework

#59

Post by brrrrrm »

orlp wrote: post #29

> It's extremely simple, and breaks down the most complex networks into 4 OpTypes: > > - UnaryOps operate on one tensor and run elementwise. RELU, LOG, RECIPROCAL, etc... > - BinaryOps operate on two tensors and run elementwise to return one. ADD, MUL, etc... > - ReduceOps operate on one tensor and return a smaller tensor. SUM, MAX > - MovementOps operate on one tensor and move the data around, copy-free with ShapeTr…

I've done some work in the past on NN representations and you actually can represent Conv and MatMul in more primitive ways. I ended up writing an IR called loop_tool that exposes this stuff:

https://github.com/facebookresearch/loop_tool/blob/main/pyth...

The idea is basically this: https://news.ycombinator.com/item?id=28883086

Post reply on HN