Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Tinygrad: A simple and powerful neural network framework

tinygrad.org

Post reply on HN
Print view View original

121–130 of 147 posts

Re: Tinygrad: A simple and powerful neural network framework

#121

Post by brrrrrm »

thrtythreeforty wrote: post #118
brrrrrm wrote: post #106

Earlier quoted context omitted.

avoiding writes to memory and reducing the number of loops (although not FLOPs) for j in range(10): c[j] = a[j] + b[j] for j in range(10): d[j] = c[j] * 2 becomes for j in range(10): d[j] = (a[j] + b[j]) * 2

Or, better, identifying that the machine has a primitive that is better than doing each op individually. For example, a multiply-accumulate instruction vs a multiply and separate accumulate. The source code still says "a*b+c", the compiler is just expected to infer the MAC instruction.

Yep! This is an assumed optimization when it comes to modern linear algebra compilers. New primitives go way beyond FMAs: full matrix multiplies on nvidia/Intel and outer product accumulates on Apple silicon. It’s also expected that these are used nearly optimally (or you’ve got a bug).

Re: Tinygrad: A simple and powerful neural network framework

#122

Post by zorgmonkey »

terafo wrote: post #113
georgehotz wrote: post #107

Earlier quoted context omitted.

Agreed. For anything at all common, most of the gains will be from fusion, the rest is just free. PyTorch also uses tons of GPU memory after only initializing, I wonder if it's copying all the kernels in?

Jax preallocates 90% of available GPU memory when first operation is run to minimize allocation overhead. Can PyTorch grab that VRAM for a similar reason?

Yes PyTorch uses what they call a caching memory allocator[0], basically seems like are allocating a very chunk of GPU memory and implementing a heap with it. If needed they expose some knobs and functions to allow you to control it and observe the memory usage.

[0]: https://pytorch.org/docs/stable/notes/cuda.html#memory-manag...

Re: Tinygrad: A simple and powerful neural network framework

#123

Post by gamegoblin »

kwant_kiddo wrote: post #120

I think posts like this are only getting upvotes because George Hotz owns the project. I do see value in simple code, but the constraint of 1000 LOC makes little sense to me, especially when the code is formatted poorly. This will get downvoted, but reading the comments here I dont understand the (cult/respect) for him. Siding with the most successful CTF-team ever (PPP) he won defcon two times. He made a startup wit…

Your list lacks the reason for his initial fame: iPhone and PS3 jailbreaking.

And I think you're downplaying the achievements of Comma AI -- it may still be somewhat niche, but its product is better than Tesla Autopilot for highway driving (they aren't there on city driving / FSD yet), all with an absolutely tiny team.

Re: Tinygrad: A simple and powerful neural network framework

#124

Post by lostmsu »

terafo wrote: post #116
lostmsu wrote: post #115

Earlier quoted context omitted.

> Actual code of tinygrad is less than 5k lines Yeah, not > Considering the code style I mean it is possible to read it, but I would not say it is optimized for it. Which I suppose betrays the goal.

> Yeah, not Provide some evidence. I just ran tokei on freshly cloned tinygrad repo using arguments from[1]. Got 4854 lines of code, which is less than 5k lines. [1] tokei --exclude *.json --exclude accel/cherry --exclude test --exclude examples

After running black it is more like 6.8k. And black does not format C and/or shader code.

But even 5k is closer to 20k on the log scale than to the promised 1k.

Re: Tinygrad: A simple and powerful neural network framework

#125

Post by gamegoblin »

tucosan wrote: post #88

Can someone from the ML crowd ELI5 to me what tinygrad does, how it plugs into an ML pipeline and what it's use cases are?

There are libraries like tensorflow and PyTorch that allow the user to define their neural net in simple, readable Python code, and they internally "compile" and optimize your neural net to run on GPUs and such.

Tinygrad is like a very, very lean PyTorch with a different philosophy -- it intends to keep the codebase and API surface very very small and focus most of its energy on optimizing the way the output neural net runs on physical hardware.

The author, George Hotz, has observed in the last few years that neural net performance is hindered by lack of optimization here, particularly around memory accesses.

Re: Tinygrad: A simple and powerful neural network framework

#126

Post by twothreeone »

georgehotz wrote: post #97
brrrrrm wrote: post #61

> It compiles a custom kernel for every operation, allowing extreme shape specialization. This doesn't matter. Just look at the performance achieved by CuDNN kernels (which back PyTorch), they're dynamically shaped and hit near peak. For dense linear algebra at the size of modern neural networks, optimizing for the loop bound condition won't help much. > All tensors are lazy, so it can aggressively fuse operations. T…

> they're dynamically shaped and hit near peak While this is true for most common GEMM looking ops, if you tread off the beaten path things get slow (odd channel sizes, batch sizes, etc...). Right now in PyTorch, GroupNorm is 2x slower than BatchNorm. There's no fundamental reason, just that the kernels loop over axes in a less than ideal order. Dynamic recompilation allows you to change the loop order too, not just…

> Right now in PyTorch, GroupNorm is 2x slower than BatchNorm

How did you benchmark this? I think there are like 3 or 4 different GN implementations in PyTorch..

Re: Tinygrad: A simple and powerful neural network framework

#127

Post by kurisufag »

dedoussis wrote: post #45

It's funny that geohot/tinygrad chooses to not meet the PEP8 standards [0] just to stay on brand ( [0] https://peps.python.org/pep-0008/ [1] https://github.com/psf/black

to anybody experienced in writing functional-esque oneliners, PEP8 is an appalling waste of space

Re: Tinygrad: A simple and powerful neural network framework

#128

Post by lostmsu »

gamegoblin wrote: post #123
kwant_kiddo wrote: post #120

I think posts like this are only getting upvotes because George Hotz owns the project. I do see value in simple code, but the constraint of 1000 LOC makes little sense to me, especially when the code is formatted poorly. This will get downvoted, but reading the comments here I dont understand the (cult/respect) for him. Siding with the most successful CTF-team ever (PPP) he won defcon two times. He made a startup wit…

Your list lacks the reason for his initial fame: iPhone and PS3 jailbreaking. And I think you're downplaying the achievements of Comma AI -- it may still be somewhat niche, but its product is better than Tesla Autopilot for highway driving (they aren't there on city driving / FSD yet), all with an absolutely tiny team.

Re: Comma AI. This is what it tells me about my run-of-the-mill Toyota:

> openpilot upgrades your Toyota Highlander Hybrid with automated lane centering at all speeds, and adaptive cruise control that automatically resumes from a stop.

Both are annoying artificial limitations Toyota put presumably to avoid abuse by inattentive drivers.

I mean it can't change lanes. What does it do exactly?

Re: Tinygrad: A simple and powerful neural network framework

#129

Post by gamegoblin »

lostmsu wrote: post #128
gamegoblin wrote: post #123

Earlier quoted context omitted.

Your list lacks the reason for his initial fame: iPhone and PS3 jailbreaking. And I think you're downplaying the achievements of Comma AI -- it may still be somewhat niche, but its product is better than Tesla Autopilot for highway driving (they aren't there on city driving / FSD yet), all with an absolutely tiny team.

Re: Comma AI. This is what it tells me about my run-of-the-mill Toyota: > openpilot upgrades your Toyota Highlander Hybrid with automated lane centering at all speeds, and adaptive cruise control that automatically resumes from a stop. Both are annoying artificial limitations Toyota put presumably to avoid abuse by inattentive drivers. I mean it can't change lanes. What does it do exactly?

Comma AI deliberately made lane change require a small bit of human intervention for safety reasons. The human hits the blinker and gives the wheel a tiny nudge in the direction, and then openpilot will complete the lane change and resume driving in the new lane.

The theory is that at the current ability of software like Tesla and Comma has, it's probably a good idea for a human to be paying more attention during a lane change maneuver. Comma is of the opinion that the level of autonomy Teslas have is probably unnecessarily unsafe. Comma cares a lot about safety (e.g. they have much more sophisticated driver monitoring than Tesla).

Lane change here: https://www.youtube.com/shorts/xm8DRwvLObQ

Since openpilot is open source software, there are of course forks that exist that remove these safety limitations and will lane change automatically.

Re: Tinygrad: A simple and powerful neural network framework

#130

Post by bfrankline »

If you care exclusively about numerical stability and performance, why _this_ set of operators (e.g., there’re plenty of good reasons to include expm1 or log1p and certainly trigonometric functions)? It’d be an interesting research problem to measure and identify the minimal subset of operators (and I suspect it’d look differently than what you’d expect from an FPU).

If you care exclusively about minimalism, why not limit yourself to the Meijer-G function (or some other general-purpose alternative)?

Post reply on HN