TinyTinyTPU: 2×2 systolic-array TPU-style matrix-multiply unit deployed on FPGA
1–10 of 61 posts
Re: TinyTinyTPU: 2×2 systolic-array TPU-style matrix-multiply unit deployed on FPGA
#2I think I could trust AI more if we used it to do heuristics for expensive deterministic processes. Sort of a cross between Bloom Filters and speculative execution. Determine the odds the expensive operation 1 will indicate that expensive operation 2 needs to happen, and then start expensive operation 2 while we determine if it’s actually needed. If its right 95% of the time, which is the sort of ranges AI can aspire to, that’s skipping the high latency task chaining 19 times out of 20, which would be pretty good.
Re: TinyTinyTPU: 2×2 systolic-array TPU-style matrix-multiply unit deployed on FPGA
#3I think it’s only a matter of time before we see asic vendors making TPU devices. Same thing happened with BTC. There was enough money there to spawn an industry. Nvidias 70% margins are too hard to ignore. And if playing on the open market seems too rough, there’s always acquisition potential like what happened to groq.
Re: TinyTinyTPU: 2×2 systolic-array TPU-style matrix-multiply unit deployed on FPGA
#4I think I could trust AI more if we used it to do heuristics for expensive deterministic processes. Sort of a cross between Bloom Filters and speculative execution. Determine the odds the expensive operation 1 will indicate that expensive operation 2 needs to happen, and then start expensive operation 2 while we determine if it’s actually needed. If its right 95% of the time, which is the sort of ranges AI can aspire…
There have been comments that some leading AI researchers were switching away from working on language models to do stuff with "real world data".
Re: TinyTinyTPU: 2×2 systolic-array TPU-style matrix-multiply unit deployed on FPGA
#5Great! How do you program it?
Re: TinyTinyTPU: 2×2 systolic-array TPU-style matrix-multiply unit deployed on FPGA
#6I think it’s only a matter of time before we see asic vendors making TPU devices. Same thing happened with BTC. There was enough money there to spawn an industry. Nvidias 70% margins are too hard to ignore. And if playing on the open market seems too rough, there’s always acquisition potential like what happened to groq.
Aren't high end accelerators already closer to ASICs than to og GPUs, tho?
Re: TinyTinyTPU: 2×2 systolic-array TPU-style matrix-multiply unit deployed on FPGA
#7I think I could trust AI more if we used it to do heuristics for expensive deterministic processes. Sort of a cross between Bloom Filters and speculative execution. Determine the odds the expensive operation 1 will indicate that expensive operation 2 needs to happen, and then start expensive operation 2 while we determine if it’s actually needed. If its right 95% of the time, which is the sort of ranges AI can aspire…
There are Bayesian neural networks that could apparently track probability rather than just e.g. randomly selecting one output from the top-k based on probability, but I'm still learning up on them myself. Sounds like they're not normally combined with language models.
Re: TinyTinyTPU: 2×2 systolic-array TPU-style matrix-multiply unit deployed on FPGA
#8Such a cool project! Next one is to run jaxprs via the driver?
Re: TinyTinyTPU: 2×2 systolic-array TPU-style matrix-multiply unit deployed on FPGA
#9I've been wondering when we will see general purpose consumer FPGAs, and eventually ASICs, for inference. This reminds me of bitcoin mining. Bitcoin mining started with GPUs. I think I remember a brief FPGA period that transitioned to ASIC. My limited understanding of Google's tensor processing unit chips are that they are effectively a transformer ASIC. That's likely a wild over-simplification of Google's TPU, but Gemini is proof that GPUs are not needed for inference.
I suspect GPU inference will come to an end soon, as it will likely be wildly inefficient by comparison to purpose built transformer chips. All those Nvidia GPU-based servers may become obsolete should transformer ASICs become mainstream. GPU bitcoin mining is just an absolute waste of money (cost of electricity) now. I believe the same will be true for GPU-based inference soon. The hundreds of billions of dollars being invested on GPU-based inference seems like an extremely risky bet that ASIC transformers won't happen, although Google has already widely deployed their own TPUs.
Re: TinyTinyTPU: 2×2 systolic-array TPU-style matrix-multiply unit deployed on FPGA
#10This is cool. I'm observing a trend of "build a tiny version from the ground-up to understand it" a la Karpathy's micrograd/minGPT. Seems like one of the best ways to learn.