> It compiles a custom kernel for every operation, allowing extreme shape specialization. This doesn't matter. Just look at the performance achieved by CuDNN kernels (which back PyTorch), they're dynamically shaped and hit near peak. For dense linear algebra at the size of modern neural networks, optimizing for the loop bound condition won't help much. > All tensors are lazy, so it can aggressively fuse operations. T…
Any more writing on laziness in frameworks? I'm trying to implement it myself.
Tinygrad: A simple and powerful neural network framework
91–100 of 147 posts
Re: Tinygrad: A simple and powerful neural network framework
#92> It's extremely simple, and breaks down the most complex networks into 4 OpTypes: > > - UnaryOps operate on one tensor and run elementwise. RELU, LOG, RECIPROCAL, etc... > - BinaryOps operate on two tensors and run elementwise to return one. ADD, MUL, etc... > - ReduceOps operate on one tensor and return a smaller tensor. SUM, MAX > - MovementOps operate on one tensor and move the data around, copy-free with ShapeTr…
That CONV is only used on the older backends. The GPU and LLVM backend rewrite CONV as MUL+SUM, to be fused later, and thus only use the 4 OpTypes.
https://github.com/geohot/tinygrad/blob/master/tinygrad/lazy...
Re: Tinygrad: A simple and powerful neural network framework
#93> It's extremely simple, and breaks down the most complex networks into 4 OpTypes: > > - UnaryOps operate on one tensor and run elementwise. RELU, LOG, RECIPROCAL, etc... > - BinaryOps operate on two tensors and run elementwise to return one. ADD, MUL, etc... > - ReduceOps operate on one tensor and return a smaller tensor. SUM, MAX > - MovementOps operate on one tensor and move the data around, copy-free with ShapeTr…
To directly quote the source: # these are the llops your accelerator must implement, along with toCpu UnaryOps = Enum("UnaryOps", ["NOOP", "NEG", "RELU", "EXP", "LOG", "SIGN", "RECIPROCAL"]) BinaryOps = Enum("BinaryOps", ["ADD", "SUB", "MUL", "DIV", "POW", "CMPEQ"]) ReduceOps = Enum("ReduceOps", ["SUM", "MAX"]) MovementOps = Enum("MovementOps", ["RESHAPE", "PERMUTE", "EXPAND", "FLIP", "STRIDED", "PAD", "SHRINK"]) Pro…
Min is an HLOP.
From: https://github.com/geohot/tinygrad/blob/master/tinygrad/tens...
def min(self, axis=None, keepdim=False): return -((-self).max(axis=axis, keepdim=keepdim))
All folded together, no slower than MAX.
Re: Tinygrad: A simple and powerful neural network framework
#94Earlier quoted context omitted.
To directly quote the source: # these are the llops your accelerator must implement, along with toCpu UnaryOps = Enum("UnaryOps", ["NOOP", "NEG", "RELU", "EXP", "LOG", "SIGN", "RECIPROCAL"]) BinaryOps = Enum("BinaryOps", ["ADD", "SUB", "MUL", "DIV", "POW", "CMPEQ"]) ReduceOps = Enum("ReduceOps", ["SUM", "MAX"]) MovementOps = Enum("MovementOps", ["RESHAPE", "PERMUTE", "EXPAND", "FLIP", "STRIDED", "PAD", "SHRINK"]) Pro…
Min is an HLOP. From: https://github.com/geohot/tinygrad/blob/master/tinygrad/tens... def min(self, axis=None, keepdim=False): return -((-self).max(axis=axis, keepdim=keepdim)) All folded together, no slower than MAX.
But then SUB, DIV and RELU could be an HLOP as well, no?
Re: Tinygrad: A simple and powerful neural network framework
#95> It compiles a custom kernel for every operation, allowing extreme shape specialization. This doesn't matter. Just look at the performance achieved by CuDNN kernels (which back PyTorch), they're dynamically shaped and hit near peak. For dense linear algebra at the size of modern neural networks, optimizing for the loop bound condition won't help much. > All tensors are lazy, so it can aggressively fuse operations. T…
What does it mean to "fuse operations"?
Re: Tinygrad: A simple and powerful neural network framework
#96It was ok as an educational tool, but now they don't count GPU implementation in 1000 lines, so it is not small. Considering the code style it is closer to 20k+ lines when formatted and GPU code included.
It also doesn't support bfloat16 so is doomed to be 2x slower.
Re: Tinygrad: A simple and powerful neural network framework
#97> It compiles a custom kernel for every operation, allowing extreme shape specialization. This doesn't matter. Just look at the performance achieved by CuDNN kernels (which back PyTorch), they're dynamically shaped and hit near peak. For dense linear algebra at the size of modern neural networks, optimizing for the loop bound condition won't help much. > All tensors are lazy, so it can aggressively fuse operations. T…
> they're dynamically shaped and hit near peak
While this is true for most common GEMM looking ops, if you tread off the beaten path things get slow (odd channel sizes, batch sizes, etc...). Right now in PyTorch, GroupNorm is 2x slower than BatchNorm. There's no fundamental reason, just that the kernels loop over axes in a less than ideal order. Dynamic recompilation allows you to change the loop order too, not just deal with boundary conditions.
Re: Tinygrad: A simple and powerful neural network framework
#98> It's extremely simple, and breaks down the most complex networks into 4 OpTypes: > > - UnaryOps operate on one tensor and run elementwise. RELU, LOG, RECIPROCAL, etc... > - BinaryOps operate on two tensors and run elementwise to return one. ADD, MUL, etc... > - ReduceOps operate on one tensor and return a smaller tensor. SUM, MAX > - MovementOps operate on one tensor and move the data around, copy-free with ShapeTr…
That CONV is only used on the older backends. The GPU and LLVM backend rewrite CONV as MUL+SUM, to be fused later, and thus only use the 4 OpTypes. https://github.com/geohot/tinygrad/blob/master/tinygrad/lazy...
That's cool, am I right in assuming that you want to automate the production of efficient GPU (or other accelerator) code based on these low level primitives? But you would still need a piece of sorcery that can produce high performance OpenCL code, right? And that code could be different for every device, so you would need some trial and error, benchmark-based compilation at the very least. Or would OpenCL code be generated by hand for each device?
Re: Tinygrad: A simple and powerful neural network framework
#99Earlier quoted context omitted.
Min is an HLOP. From: https://github.com/geohot/tinygrad/blob/master/tinygrad/tens... def min(self, axis=None, keepdim=False): return -((-self).max(axis=axis, keepdim=keepdim)) All folded together, no slower than MAX.
But then SUB, DIV and RELU could be an HLOP as well, no?
We could have NEG instead of SUB, but with the constant folding it's a wash. DIV is already an HLOP with reciprocal (used to use POW, but that was slower. And what would you implement RELU in terms of?
Re: Tinygrad: A simple and powerful neural network framework
#100Earlier quoted context omitted.
It definitely works, JAX only sees the unrolled loop: x = 0 x += y x += y x += y x += y x += y return x The reason you might need `jax.lax.fori_loop` or some such is if you have a long loop with a complex body. Replicating a complex body many times means you end up with a huge computation graph and slow compilation.
And how does TinyGrad solve this?
Fused into one operation since the Tensor isn't resolved until I call .numpy()
kafka@tubby:/tmp$ cat fuse.py
from tinygrad.tensor import Tensor
x = Tensor.zeros(1)
for i in range(5):
x += i
print(x.numpy())
kafka@tubby:/tmp$ OPT=2 GPU=1 DEBUG=2 python3 fuse.py
using []
**CL** 0 elementwise_0 args 1 kernels [1, 1, 1] None OPs 0.0M/ 0.00G mem 0.00 GB tm 0.15us/ 0.00ms ( 0.03 GFLOPS)
**CL** copy OUT (1,)
[10.]