Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Tinygrad: A simple and powerful neural network framework

tinygrad.org

Post reply on HN
Print view View original

31–40 of 147 posts

Re: Tinygrad: A simple and powerful neural network framework

#31

Post by jstx1 »

DeathArrow wrote: post #3

I believe neural networks are over hyped sometimes. They are not always the best tool for the job. There are lots of other ML techniques such as SVM, naive Bayes, k-nearest neighbor, decision tree, logistic regression, random forest etc. nobody is using because they lack the hype factor. If something lacks some keywords like neural network, deep learning, reinforced learning, than it is deemed not cool.

Everything you're listing works mostly on tabular data, not on text or images which is where we have the most impressive ML applications right now.

Re: Tinygrad: A simple and powerful neural network framework

#32

Post by jjallen »

matesz wrote: post #9

If anybody is dealing with procrastination watch George Hotz live streaming 10h straight working on this library [1][2]. Does he take some supplements to do this? There is even 19.5h stream [3]. Actually I have local obs setup to record myself, just instead of streaming I do recordings for my own inspection. Important part is to do the inspection after. It works wonders. [1] https://youtu.be/GXy5eVwnL_Q [2] https://m…

What do you do with your own recordings afterwards? How do they help you?

Re: Tinygrad: A simple and powerful neural network framework

#33

Post by eterevsky »

How is it compared to JAX? After TensorFlow and PyTorch, JAX seems very simple, basically an accelerated numpy with just a few additional useful features like automatic differentiation, vectorization and jit-compilation. In terms of API I don't see how you can go any simpler.

Re: Tinygrad: A simple and powerful neural network framework

#34

Post by langsoul-com »

matesz wrote: post #9

If anybody is dealing with procrastination watch George Hotz live streaming 10h straight working on this library [1][2]. Does he take some supplements to do this? There is even 19.5h stream [3]. Actually I have local obs setup to record myself, just instead of streaming I do recordings for my own inspection. Important part is to do the inspection after. It works wonders. [1] https://youtu.be/GXy5eVwnL_Q [2] https://m…

What's the file size for your recordings? 5 hours of 720p would be huge.

Re: Tinygrad: A simple and powerful neural network framework

#36

Post by WithinReason »

orlp wrote: post #29

> It's extremely simple, and breaks down the most complex networks into 4 OpTypes: > > - UnaryOps operate on one tensor and run elementwise. RELU, LOG, RECIPROCAL, etc... > - BinaryOps operate on two tensors and run elementwise to return one. ADD, MUL, etc... > - ReduceOps operate on one tensor and return a smaller tensor. SUM, MAX > - MovementOps operate on one tensor and move the data around, copy-free with ShapeTr…

To directly quote the source:

    # these are the llops your accelerator must implement, along with toCpu
    UnaryOps = Enum("UnaryOps", ["NOOP", "NEG", "RELU", "EXP", "LOG", "SIGN", "RECIPROCAL"])
    BinaryOps = Enum("BinaryOps", ["ADD", "SUB", "MUL", "DIV", "POW", "CMPEQ"])
    ReduceOps = Enum("ReduceOps", ["SUM", "MAX"])
    MovementOps = Enum("MovementOps", ["RESHAPE", "PERMUTE", "EXPAND", "FLIP", "STRIDED", "PAD", "SHRINK"])
    ProcessingOps = Enum("ProcessingOps", ["CONV"])
https://github.com/geohot/tinygrad/blob/caea34c52996cde2ed46...

There is a MAX but not a MIN? Is that because max(x,y) = -min(-x,-y)? But then why is there a SUB? Why is there a RELU if it's only max(0,x)? Maybe MIN is just too rare to be worth implementing?

Re: Tinygrad: A simple and powerful neural network framework

#38

Post by albert_e »

langsoul-com wrote: post #34
matesz wrote: post #9

If anybody is dealing with procrastination watch George Hotz live streaming 10h straight working on this library [1][2]. Does he take some supplements to do this? There is even 19.5h stream [3]. Actually I have local obs setup to record myself, just instead of streaming I do recordings for my own inspection. Important part is to do the inspection after. It works wonders. [1] https://youtu.be/GXy5eVwnL_Q [2] https://m…

What's the file size for your recordings? 5 hours of 720p would be huge.

YouTube lets you livestream from OBS but mark the stream as private.

You get unlimited free storage of your streams for your personal use that way without the need for any local storage at all.

I haven't come across any limits or downsides to this yet but happy to be corrected.

Re: Tinygrad: A simple and powerful neural network framework

#39

Post by cl3misch »

eterevsky wrote: post #33

How is it compared to JAX? After TensorFlow and PyTorch, JAX seems very simple, basically an accelerated numpy with just a few additional useful features like automatic differentiation, vectorization and jit-compilation. In terms of API I don't see how you can go any simpler.

He mentioned in a recent stream that he dislikes the complexity of the XLA instruction set used by JAX. So it's less the user-facing API, and more the inner workings of the library.

Re: Tinygrad: A simple and powerful neural network framework

#40

Post by learndeeply »

eterevsky wrote: post #33

How is it compared to JAX? After TensorFlow and PyTorch, JAX seems very simple, basically an accelerated numpy with just a few additional useful features like automatic differentiation, vectorization and jit-compilation. In terms of API I don't see how you can go any simpler.

JAX is a DSL on top of XLA, instead of writing Python. Example: a JAX for loop looks like this:

   def summ(i, v): return i + v
   x = jax.lax.fori_loop(0, 100, summ, 5)
A for loop in TinyGrad or PyTorch looks like regular Python:

   x = 5
   for i in range(0, 100):
      x += 1
By the way, PyTorch also has JIT.
Post reply on HN