Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

TinyTinyTPU: 2×2 systolic-array TPU-style matrix-multiply unit deployed on FPGA

github.com

Post reply on HN
Print view View original

51–60 of 61 posts

Re: TinyTinyTPU: 2×2 systolic-array TPU-style matrix-multiply unit deployed on FPGA

#51

Post by imtringued »

fooblaster wrote: post #12
mrinterweb wrote: post #9

I've been wondering when we will see general purpose consumer FPGAs, and eventually ASICs, for inference. This reminds me of bitcoin mining. Bitcoin mining started with GPUs. I think I remember a brief FPGA period that transitioned to ASIC. My limited understanding of Google's tensor processing unit chips are that they are effectively a transformer ASIC. That's likely a wild over-simplification of Google's TPU, but G…

FPGAs will never rival gpus or TPUs for inference. The main reason is that GPUs aren't really gpus anymore. 50% of the die area or more is for fixed function matrix multiplication units and associated dedicated storage. This just isn't general purpose anymore. FPGAs cannot rival this with their configurable DSP slices. They would need dedicated systolic blocks, which they aren't getting. The closest thing is the vers…

I feel like your entire comment is a self contradicting mess.

You say FPGAs won't get dedicated logic for ML, then you say they did.

Why does it matter whether the matrix multiplication units inside the AI Engine are a systolic array or not? The multipliers support 512 bit inputs which means 4x8 times 8x4 for bfloat16 with one multiplication per cycle and bigger multiplications with smaller data types. Since it is a VLIW processor, it is much easier to achieve full utilisation of the matrix multiplication units, because you can run loads, stores and process tiles all simultaneously in the same cycle.

The only thing that might be a challenge is arranging the communication between the AI Engines, but even that should be blatantly obvious. If you are doing matrix multiplication, you should be using the entire array in exactly the pattern you think they should be using internally.

Who knows, maybe there is a way to implement flash attention like that too.

Re: TinyTinyTPU: 2×2 systolic-array TPU-style matrix-multiply unit deployed on FPGA

#52

Post by seamossfet »

mrinterweb wrote: post #9

I've been wondering when we will see general purpose consumer FPGAs, and eventually ASICs, for inference. This reminds me of bitcoin mining. Bitcoin mining started with GPUs. I think I remember a brief FPGA period that transitioned to ASIC. My limited understanding of Google's tensor processing unit chips are that they are effectively a transformer ASIC. That's likely a wild over-simplification of Google's TPU, but G…

The only time FPGAs / ASICS are better is if there's gains we can make by innovating on the hardware architecture itself. That's pretty hard to do considering GPUs are already heavily optimized for this use case.

Re: TinyTinyTPU: 2×2 systolic-array TPU-style matrix-multiply unit deployed on FPGA

#53

Post by hinkley »

0-_-0 wrote: post #50
hinkley wrote: post #2

I think I could trust AI more if we used it to do heuristics for expensive deterministic processes. Sort of a cross between Bloom Filters and speculative execution. Determine the odds the expensive operation 1 will indicate that expensive operation 2 needs to happen, and then start expensive operation 2 while we determine if it’s actually needed. If its right 95% of the time, which is the sort of ranges AI can aspire…

CPU branch predictors use neural networks

CPU branch predictors aren’t going to run long expensive operations in the background. This is like saying bloom filters are speculative memory fetches. That’s not completely untrue but it misses the point.

Re: TinyTinyTPU: 2×2 systolic-array TPU-style matrix-multiply unit deployed on FPGA

#54

Post by fooblaster »

dnautics wrote: post #46
fooblaster wrote: post #44

Earlier quoted context omitted.

Yep, but they are still 50x faster than any fpga.

probably not B200 level but better than you might expect: https://www.positron.ai/ i believe a B200 is ~3x the H200 at llama-3, so that puts the FPGAs at around 60% the speed of B200s?

I wouldn't trust any benchmarks on the vendors site. Microsoft went down this path for years with FPGAs and wrote off the entire effort.

Re: TinyTinyTPU: 2×2 systolic-array TPU-style matrix-multiply unit deployed on FPGA

#55

Post by dnautics »

fooblaster wrote: post #54
dnautics wrote: post #46

Earlier quoted context omitted.

probably not B200 level but better than you might expect: https://www.positron.ai/ i believe a B200 is ~3x the H200 at llama-3, so that puts the FPGAs at around 60% the speed of B200s?

I wouldn't trust any benchmarks on the vendors site. Microsoft went down this path for years with FPGAs and wrote off the entire effort.

ok? i worked on those devices, those numbers are real. theres a reason why they compare to h200 and not b200

> I have worked with FPGAs that outperform H200s in Llama3-class models a while and a half ago

Re: TinyTinyTPU: 2×2 systolic-array TPU-style matrix-multiply unit deployed on FPGA

#56

Post by teleforce »

fooblaster wrote: post #12
mrinterweb wrote: post #9

I've been wondering when we will see general purpose consumer FPGAs, and eventually ASICs, for inference. This reminds me of bitcoin mining. Bitcoin mining started with GPUs. I think I remember a brief FPGA period that transitioned to ASIC. My limited understanding of Google's tensor processing unit chips are that they are effectively a transformer ASIC. That's likely a wild over-simplification of Google's TPU, but G…

FPGAs will never rival gpus or TPUs for inference. The main reason is that GPUs aren't really gpus anymore. 50% of the die area or more is for fixed function matrix multiplication units and associated dedicated storage. This just isn't general purpose anymore. FPGAs cannot rival this with their configurable DSP slices. They would need dedicated systolic blocks, which they aren't getting. The closest thing is the vers…

>Those have failed by being impossible to program.

I think you spoke too soon about their failure, sooner they will be much easier to program [1].

Interestingly, Nvidia GPU now is also moving to tile-based GPU programming model that targets portability for NVIDIA Tensor Cores [2]. Recently there're discussions on the topic at HN [3].

[1] Developing a BLAS Library for the AMD AI Engine [pdf]:

https://uni.tlaan.nl/thesis/msc_thesis_tristan_laan_aieblas....

[2] NVIDIA CUDA Tile:

https://developer.nvidia.com/cuda/tile

[3]CUDA Tile Open Sourced (103 comments):

https://news.ycombinator.com/item?id=46330732

Re: TinyTinyTPU: 2×2 systolic-array TPU-style matrix-multiply unit deployed on FPGA

#57

Post by fooblaster »

teleforce wrote: post #56
fooblaster wrote: post #12

Earlier quoted context omitted.

FPGAs will never rival gpus or TPUs for inference. The main reason is that GPUs aren't really gpus anymore. 50% of the die area or more is for fixed function matrix multiplication units and associated dedicated storage. This just isn't general purpose anymore. FPGAs cannot rival this with their configurable DSP slices. They would need dedicated systolic blocks, which they aren't getting. The closest thing is the vers…

>Those have failed by being impossible to program. I think you spoke too soon about their failure, sooner they will be much easier to program [1]. Interestingly, Nvidia GPU now is also moving to tile-based GPU programming model that targets portability for NVIDIA Tensor Cores [2]. Recently there're discussions on the topic at HN [3]. [1] Developing a BLAS Library for the AMD AI Engine [pdf]: https://uni.tlaan.nl/thes…

The amd npu and versal ML tiles (same underlying architecture) have been an complete failure. Dynamic programming models like cu tile do not work on them at all, be cause they require an entirely static graph to function. AMD is going to walk away from their NPU architecture and unify around their GPU IP on inference products in the future.

Re: TinyTinyTPU: 2×2 systolic-array TPU-style matrix-multiply unit deployed on FPGA

#58

Post by fooblaster »

dnautics wrote: post #55
fooblaster wrote: post #54

Earlier quoted context omitted.

I wouldn't trust any benchmarks on the vendors site. Microsoft went down this path for years with FPGAs and wrote off the entire effort.

ok? i worked on those devices, those numbers are real. theres a reason why they compare to h200 and not b200 > I have worked with FPGAs that outperform H200s in Llama3-class models a while and a half ago

I'd like to know more. I expect these systems are 8xvh1782. Is that true? What's the theoretical math throughput - my expectation is that it isn't very high per chip. How is performance in the prefill stage when inference is actually math limited?

Re: TinyTinyTPU: 2×2 systolic-array TPU-style matrix-multiply unit deployed on FPGA

#59

Post by mrinterweb »

zhemao wrote: post #45
mrinterweb wrote: post #9

I've been wondering when we will see general purpose consumer FPGAs, and eventually ASICs, for inference. This reminds me of bitcoin mining. Bitcoin mining started with GPUs. I think I remember a brief FPGA period that transitioned to ASIC. My limited understanding of Google's tensor processing unit chips are that they are effectively a transformer ASIC. That's likely a wild over-simplification of Google's TPU, but G…

TPUs aren't transformer ASICs. The Ironwood TPU that Gemini was trained on was designed before LLMs became popular with ChatGPT's release. The architecture was general enough that it ended up being efficient for LLM training. A special-purpose transformer inference ASIC would be like Etched's Sohu chip.

> TPUs aren't transformer ASICs.

https://cloud.google.com/tpu

> A TPU is an application-specific integrated circuit (ASIC) designed by Google for neural networks.

Re: TinyTinyTPU: 2×2 systolic-array TPU-style matrix-multiply unit deployed on FPGA

#60

Post by dnautics »

fooblaster wrote: post #58
dnautics wrote: post #55

Earlier quoted context omitted.

ok? i worked on those devices, those numbers are real. theres a reason why they compare to h200 and not b200 > I have worked with FPGAs that outperform H200s in Llama3-class models a while and a half ago

I'd like to know more. I expect these systems are 8xvh1782. Is that true? What's the theoretical math throughput - my expectation is that it isn't very high per chip. How is performance in the prefill stage when inference is actually math limited?

i was a software guy, sorry, but those token rates are correct and what was flowing through my software.

i believe there was a special deal on super special fpgas. there were dsps involved.

Post reply on HN