101–110 of 147 posts
Earlier quoted context omitted.
But then SUB, DIV and RELU could be an HLOP as well, no?
We could have NEG instead of SUB, but with the constant folding it's a wash. DIV is already an HLOP with reciprocal (used to use POW, but that was slower. And what would you implement RELU in terms of?
I understand that the Python code is mostly driving faster low-level code, but I wonder how much time is effectively wasted by not using a lower-level language. From my experience with game engines, it often turns out to be a bad idea (for performance and maintainability) to mix C/C++ and Lua or C#.
I would argue that there are performance *benefits* for a developer in running python code, due to how programs are run in python(Jupyter notebooks) you basically can change program on the fly, and not recompile and restart it, as you would do with compiled languages. And yeah, CPU does very very little in modern DL workloads and it is commonplace for CPU python code to be jitted and vectorized, so performance differ…
Another benefit to interactivity is when exploring/using bad code. In academia, you'll often be importing the worst and least-well-documented code you've ever seen.
Being able to interactively experiment with someones 500-line 0-documentation function is often a better path to understanding than directly reading the code.
Earlier quoted context omitted.
That CONV is only used on the older backends. The GPU and LLVM backend rewrite CONV as MUL+SUM, to be fused later, and thus only use the 4 OpTypes. https://github.com/geohot/tinygrad/blob/master/tinygrad/lazy...
That's cool, am I right in assuming that you want to automate the production of efficient GPU (or other accelerator) code based on these low level primitives? But you would still need a piece of sorcery that can produce high performance OpenCL code, right? And that code could be different for every device, so you would need some trial and error, benchmark-based compilation at the very least. Or would OpenCL code be g…
Working on parameterizing a search space that includes more than the local group size. The end dream is some ML guided search to optimize the kernels :)
> It compiles a custom kernel for every operation, allowing extreme shape specialization. This doesn't matter. Just look at the performance achieved by CuDNN kernels (which back PyTorch), they're dynamically shaped and hit near peak. For dense linear algebra at the size of modern neural networks, optimizing for the loop bound condition won't help much. > All tensors are lazy, so it can aggressively fuse operations. T…
> they're dynamically shaped and hit near peak While this is true for most common GEMM looking ops, if you tread off the beaten path things get slow (odd channel sizes, batch sizes, etc...). Right now in PyTorch, GroupNorm is 2x slower than BatchNorm. There's no fundamental reason, just that the kernels loop over axes in a less than ideal order. Dynamic recompilation allows you to change the loop order too, not just…
Yea, makes sense. I think there's something to be said for dynamic compilation solving this problem more elegantly than providing tons of hand-tuned kernels (PyTorch is 890MB lmao https://pypi.org/project/torch/#files), but I don't think it's a strict reason for a performance win.
> change the loop order too
Memory layout as well! I'm 100% for dynamic compilation, but I'm claiming that it really finds its stride when you fuse things.
Earlier quoted context omitted.
We could have NEG instead of SUB, but with the constant folding it's a wash. DIV is already an HLOP with reciprocal (used to use POW, but that was slower. And what would you implement RELU in terms of?
max(0,x)
Submit a PR if you can improve something!
> It compiles a custom kernel for every operation, allowing extreme shape specialization. This doesn't matter. Just look at the performance achieved by CuDNN kernels (which back PyTorch), they're dynamically shaped and hit near peak. For dense linear algebra at the size of modern neural networks, optimizing for the loop bound condition won't help much. > All tensors are lazy, so it can aggressively fuse operations. T…
What does it mean to "fuse operations"?
for j in range(10):
c[j] = a[j] + b[j]
for j in range(10):
d[j] = c[j] * 2
for j in range(10):
d[j] = (a[j] + b[j]) * 2Earlier quoted context omitted.
> they're dynamically shaped and hit near peak While this is true for most common GEMM looking ops, if you tread off the beaten path things get slow (odd channel sizes, batch sizes, etc...). Right now in PyTorch, GroupNorm is 2x slower than BatchNorm. There's no fundamental reason, just that the kernels loop over axes in a less than ideal order. Dynamic recompilation allows you to change the loop order too, not just…
> tread off the beaten path things get slow Yea, makes sense. I think there's something to be said for dynamic compilation solving this problem more elegantly than providing tons of hand-tuned kernels (PyTorch is 890MB lmao https://pypi.org/project/torch/#files ), but I don't think it's a strict reason for a performance win. > change the loop order too Memory layout as well! I'm 100% for dynamic compilation, but I'm…
Earlier quoted context omitted.
That's cool, am I right in assuming that you want to automate the production of efficient GPU (or other accelerator) code based on these low level primitives? But you would still need a piece of sorcery that can produce high performance OpenCL code, right? And that code could be different for every device, so you would need some trial and error, benchmark-based compilation at the very least. Or would OpenCL code be g…
Yea, benchmark based compilation, that's already happening in the tinygrad compiler we use for openpilot to determine the local group size. https://github.com/geohot/tinygrad/blob/caea34c52996cde2ed46... Working on parameterizing a search space that includes more than the local group size. The end dream is some ML guided search to optimize the kernels :)
tinygrad core is over 1000 loc now[1]. If anyone was looking for a fun weekend project :) https://github.com/geohot/tinygrad/blob/master/.github/workf...
“Almost 9k stars” is actually 7.3k stars… But otherwise very cool project :)