Tinygrad will be the next Linux and LLVM
twitter.com
Tinygrad will be the next Linux and LLVM
1–10 of 39 posts
Re: Tinygrad will be the next Linux and LLVM
#2> Without CPUs, we can be freed from the tyranny of the halting problem.
Can someone please explain to me what this even means in this context?
Serious question.
Re: Tinygrad will be the next Linux and LLVM
#3> Without CPUs, we can be freed from the tyranny of the halting problem. Can someone please explain to me what this even means in this context? Serious question.
It's gibberish. For one thing... https://arxiv.org/abs/1901.03429
Re: Tinygrad will be the next Linux and LLVM
#4I don't know why I should switch from PyTorch to Tinygrad as a researcher and practitioner. In terms of kernel fusion, there is torch.compile. Not to say there is a large ecosystem behind PyTorch and almost every paper today is published with a PyTorch implementation. Probably what Tinygrad shines is bare-metal platforms?
Re: Tinygrad will be the next Linux and LLVM
#5> tinygrad has a hardware abstraction layer, a scheduler, and memory management. It's an operating system
Doesn't every ML framework have that?
Re: Tinygrad will be the next Linux and LLVM
#6no it won't, because while hitting ioctls in python is cute
https://github.com/tinygrad/tinygrad/blob/master/extra/hip_g...
it is definitely not shippable
Re: Tinygrad will be the next Linux and LLVM
#7He's got the kernel of a good idea. Deterministic data flows are a good thing. We keep almost getting there, with things like data flow architectures, FPGAs, etc. But there's always a premature optimization for the silicon, instead of the whole system. This leads to failure, over, and over.
He's wrong in the idea of using an LLM for general purpose compute. Using math instead of logic isn't a good thing for many use cases. You don't want a database, or an FFT in a Radar System to hallucinate, for example.
My personal focus is on homogeneous, clocked, bit level systolic arrays.[2] I'm starting to get the feeling the idea is really close to being a born secret[1] though, as it might enable anyone to really make high performance chips on any fab node.
Re: Tinygrad will be the next Linux and LLVM
#8> tinygrad has a hardware abstraction layer, a scheduler, and memory management. It's an operating system Doesn't every ML framework have that?
nah not like he's talking about - TF and PT definitely punt all that down to tensorrt or hip or whatever. doesn't mean there's anything novel here - just that TF and PyTorch don't do it.
Re: Tinygrad will be the next Linux and LLVM
#9I don’t understand the LLVM comparison. Is it somehow a compiler backend for conventional programming languages? Can you run C or Rust code?
Re: Tinygrad will be the next Linux and LLVM
#10He's got the kernel of a good idea. Deterministic data flows are a good thing. We keep almost getting there, with things like data flow architectures, FPGAs, etc. But there's always a premature optimization for the silicon, instead of the whole system. This leads to failure, over, and over. He's wrong in the idea of using an LLM for general purpose compute. Using math instead of logic isn't a good thing for many use…
You could still build a FFT in tinygrad and it would be as deterministic as it's matmuls (so not bitwise deterministic, due to the non-associativity of floating point math and the way GPUs don't guarantee execution order, but we are okay with that). The matmuls in the NNs don't hallucinate.