TinyTinyTPU: 2×2 systolic-array TPU-style matrix-multiply unit deployed on FPGA

Post by fooblaster » Sat, Jan 03, 2026, 2:56 AM UTC

dnautics wrote: post #39

fooblaster wrote: post #12

Earlier quoted context omitted.

FPGAs will never rival gpus or TPUs for inference. The main reason is that GPUs aren't really gpus anymore. 50% of the die area or more is for fixed function matrix multiplication units and associated dedicated storage. This just isn't general purpose anymore. FPGAs cannot rival this with their configurable DSP slices. They would need dedicated systolic blocks, which they aren't getting. The closest thing is the vers…

I don't think this is correct. For inference, the bottleneck is memory bandwidth, so if you can hook up an FPGA with better memory, it has an outside shot at beating GPUs, at least in the short term. I mean, I have worked with FPGAs that outperform H200s in Llama3-class models a while and a half ago.

Post by checker659 » Sat, Jan 03, 2026, 5:27 AM UTC

fooblaster wrote: post #41

dnautics wrote: post #39

Earlier quoted context omitted.

I don't think this is correct. For inference, the bottleneck is memory bandwidth, so if you can hook up an FPGA with better memory, it has an outside shot at beating GPUs, at least in the short term. I mean, I have worked with FPGAs that outperform H200s in Llama3-class models a while and a half ago.

Show me a single FPGA that can outperform a B200 at matrix multiplication (or even come close) at any usable precision. B200 can do 10 peta ops at fp8, theoretically. I do agree memory bandwidth is also a problem for most FPGA setups, but xilinx ships HBM with some skus and they are not competitive at inference as far as I know.

Post by zhemao » Sat, Jan 03, 2026, 7:33 AM UTC

mrinterweb wrote: post #9

I've been wondering when we will see general purpose consumer FPGAs, and eventually ASICs, for inference. This reminds me of bitcoin mining. Bitcoin mining started with GPUs. I think I remember a brief FPGA period that transitioned to ASIC. My limited understanding of Google's tensor processing unit chips are that they are effectively a transformer ASIC. That's likely a wild over-simplification of Google's TPU, but G…

Post by hayley-patton » Sat, Jan 03, 2026, 9:03 AM UTC

beeflet wrote: post #31

fpgaminer wrote: post #28

Earlier quoted context omitted.

> FPGAs will never rival gpus or TPUs for inference. The main reason is that GPUs aren't really gpus anymore. Yeah. Even for Bitcoin mining GPUs dominated FPGAs. I created the Bitcoin mining FPGA project(s), and they were only interesting for two reasons: 1) they were far more power efficient, which in the case of mining changes the equation significantly. 2) GPUs at the time had poor binary math support, which hampe…

I have wondered if it is possible to make a mining algorithm FPGA-hard in the same way that RandomX is CPU-hard and memory-hard. Relative to CPUs, the "programming time" cost is high. Nice username btw.

Post by ksk23 » Sat, Jan 03, 2026, 10:15 AM UTC

beeflet wrote: post #31

fpgaminer wrote: post #28

Earlier quoted context omitted.

> FPGAs will never rival gpus or TPUs for inference. The main reason is that GPUs aren't really gpus anymore. Yeah. Even for Bitcoin mining GPUs dominated FPGAs. I created the Bitcoin mining FPGA project(s), and they were only interesting for two reasons: 1) they were far more power efficient, which in the case of mining changes the equation significantly. 2) GPUs at the time had poor binary math support, which hampe…

I have wondered if it is possible to make a mining algorithm FPGA-hard in the same way that RandomX is CPU-hard and memory-hard. Relative to CPUs, the "programming time" cost is high. Nice username btw.