Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

TinyTinyTPU: 2×2 systolic-array TPU-style matrix-multiply unit deployed on FPGA

github.com

Post reply on HN
Print view View original

31–40 of 61 posts

Re: TinyTinyTPU: 2×2 systolic-array TPU-style matrix-multiply unit deployed on FPGA

#31

Post by beeflet »

fpgaminer wrote: post #28
fooblaster wrote: post #12

Earlier quoted context omitted.

FPGAs will never rival gpus or TPUs for inference. The main reason is that GPUs aren't really gpus anymore. 50% of the die area or more is for fixed function matrix multiplication units and associated dedicated storage. This just isn't general purpose anymore. FPGAs cannot rival this with their configurable DSP slices. They would need dedicated systolic blocks, which they aren't getting. The closest thing is the vers…

> FPGAs will never rival gpus or TPUs for inference. The main reason is that GPUs aren't really gpus anymore. Yeah. Even for Bitcoin mining GPUs dominated FPGAs. I created the Bitcoin mining FPGA project(s), and they were only interesting for two reasons: 1) they were far more power efficient, which in the case of mining changes the equation significantly. 2) GPUs at the time had poor binary math support, which hampe…

I have wondered if it is possible to make a mining algorithm FPGA-hard in the same way that RandomX is CPU-hard and memory-hard. Relative to CPUs, the "programming time" cost is high.

Nice username btw.

Re: TinyTinyTPU: 2×2 systolic-array TPU-style matrix-multiply unit deployed on FPGA

#32

Post by mrinterweb »

nomel wrote: post #24
mrinterweb wrote: post #13

Earlier quoted context omitted.

As far as I understand all the inference purpose-build silicon out there is not being sold to competitors and kept in-house. Google's TPU, Amazon's Inferentia (horrible name), Microsoft's Maia, Meta's MTIA. It seems that custom inference silicon is a huge part of the AI game. I doubt GPU-based inference will be relevant/competitive soon.

> It seems that custom inference silicon is a huge part of the AI game. Is there any public info about % inference on custom vs GPU, for these companies?

Gemini is likely the most widely used gen AI model in the world considering search, Android integration, and countless other integrations into the Google ecosystem. Gemini runs on their custom TPU chips. So I would say a large portion of inference is already using ASIC. https://cloud.google.com/tpu

Re: TinyTinyTPU: 2×2 systolic-array TPU-style matrix-multiply unit deployed on FPGA

#33

Post by mrinterweb »

almostgotcaught wrote: post #18
mrinterweb wrote: post #13

Earlier quoted context omitted.

As far as I understand all the inference purpose-build silicon out there is not being sold to competitors and kept in-house. Google's TPU, Amazon's Inferentia (horrible name), Microsoft's Maia, Meta's MTIA. It seems that custom inference silicon is a huge part of the AI game. I doubt GPU-based inference will be relevant/competitive soon.

[flagged]

Soon was wrong. I should have said it is already happening. Google Gemini already uses their own TPU chips. Nvidia just dropped $20B to buy the IP for Groq's LPU (custom silicon for inference). $20B says Nvidia sees the writing on the wall for GPU-based inference. https://www.tomshardware.com/tech-industry/semiconductors/nv...

Re: TinyTinyTPU: 2×2 systolic-array TPU-style matrix-multiply unit deployed on FPGA

#34

Post by p1esk »

rjsw wrote: post #4
hinkley wrote: post #2

I think I could trust AI more if we used it to do heuristics for expensive deterministic processes. Sort of a cross between Bloom Filters and speculative execution. Determine the odds the expensive operation 1 will indicate that expensive operation 2 needs to happen, and then start expensive operation 2 while we determine if it’s actually needed. If its right 95% of the time, which is the sort of ranges AI can aspire…

There have been comments that some leading AI researchers were switching away from working on language models to do stuff with "real world data".

What do you mean?

Re: TinyTinyTPU: 2×2 systolic-array TPU-style matrix-multiply unit deployed on FPGA

#35

Post by [deleted] »

aunty_helen wrote: post #3

I think it’s only a matter of time before we see asic vendors making TPU devices. Same thing happened with BTC. There was enough money there to spawn an industry. Nvidias 70% margins are too hard to ignore. And if playing on the open market seems too rough, there’s always acquisition potential like what happened to groq.

[deleted]

Re: TinyTinyTPU: 2×2 systolic-array TPU-style matrix-multiply unit deployed on FPGA

#36

Post by almostgotcaught »

mrinterweb wrote: post #33
almostgotcaught wrote: post #18

Earlier quoted context omitted.

[flagged]

Soon was wrong. I should have said it is already happening. Google Gemini already uses their own TPU chips. Nvidia just dropped $20B to buy the IP for Groq's LPU (custom silicon for inference). $20B says Nvidia sees the writing on the wall for GPU-based inference. https://www.tomshardware.com/tech-industry/semiconductors/nv...

There are so many people on here that are outsiders commenting way out of their depth:

> Google Gemini already uses their own TPU chips

Google has been using TPUs in prod for like a decade.

Re: TinyTinyTPU: 2×2 systolic-array TPU-style matrix-multiply unit deployed on FPGA

#37

Post by liuliu »

mrinterweb wrote: post #9

I've been wondering when we will see general purpose consumer FPGAs, and eventually ASICs, for inference. This reminds me of bitcoin mining. Bitcoin mining started with GPUs. I think I remember a brief FPGA period that transitioned to ASIC. My limited understanding of Google's tensor processing unit chips are that they are effectively a transformer ASIC. That's likely a wild over-simplification of Google's TPU, but G…

This is a common misunderstanding from industry observers (not industry practitioners). Each generation of (NVIDIA) GPU is an ASIC with different ISA etc. Bitcoin mining simply was not important enough (last year, only $23B Bitcoin mined in total (at $100,000 per)). There is amped incentive to implement every possible instructions useful into GPU (without worrying about backward compatibility, thanks to PTX).

ASIC transformers won't happen (defined as a chip with single instruction to do sdpa from anything that is not broadly marketed as GPU, and won't have annualized sale more than $3B). Mark my word. I am happy to take a bet on longbets.org with anyone on this for $1000 and my part will go to PSF.

Re: TinyTinyTPU: 2×2 systolic-array TPU-style matrix-multiply unit deployed on FPGA

#38

Post by dnautics »

liuliu wrote: post #37
mrinterweb wrote: post #9

I've been wondering when we will see general purpose consumer FPGAs, and eventually ASICs, for inference. This reminds me of bitcoin mining. Bitcoin mining started with GPUs. I think I remember a brief FPGA period that transitioned to ASIC. My limited understanding of Google's tensor processing unit chips are that they are effectively a transformer ASIC. That's likely a wild over-simplification of Google's TPU, but G…

This is a common misunderstanding from industry observers (not industry practitioners). Each generation of (NVIDIA) GPU is an ASIC with different ISA etc. Bitcoin mining simply was not important enough (last year, only $23B Bitcoin mined in total (at $100,000 per)). There is amped incentive to implement every possible instructions useful into GPU (without worrying about backward compatibility, thanks to PTX). ASIC tr…

I don't know if they'll reach $3B, but at least one company is using FPGA transformers (that perform well) to get revenue in before going to ASIC transformers:

https://www.positron.ai/

Re: TinyTinyTPU: 2×2 systolic-array TPU-style matrix-multiply unit deployed on FPGA

#39

Post by dnautics »

fooblaster wrote: post #12
mrinterweb wrote: post #9

I've been wondering when we will see general purpose consumer FPGAs, and eventually ASICs, for inference. This reminds me of bitcoin mining. Bitcoin mining started with GPUs. I think I remember a brief FPGA period that transitioned to ASIC. My limited understanding of Google's tensor processing unit chips are that they are effectively a transformer ASIC. That's likely a wild over-simplification of Google's TPU, but G…

FPGAs will never rival gpus or TPUs for inference. The main reason is that GPUs aren't really gpus anymore. 50% of the die area or more is for fixed function matrix multiplication units and associated dedicated storage. This just isn't general purpose anymore. FPGAs cannot rival this with their configurable DSP slices. They would need dedicated systolic blocks, which they aren't getting. The closest thing is the vers…

I don't think this is correct. For inference, the bottleneck is memory bandwidth, so if you can hook up an FPGA with better memory, it has an outside shot at beating GPUs, at least in the short term.

I mean, I have worked with FPGAs that outperform H200s in Llama3-class models a while and a half ago.

Re: TinyTinyTPU: 2×2 systolic-array TPU-style matrix-multiply unit deployed on FPGA

#40

Post by ssivark »

tonetegeatinst wrote: post #17
NitpickLawyer wrote: post #6

Earlier quoted context omitted.

Aren't high end accelerators already closer to ASICs than to og GPUs, tho?

Yes, but not as much as you think. A lot of silicon on a GPU is dedicated to upscaling and matrix multiply. Ultimately GPU's main use is multimedia and graphics focused. See all the miners that used to do GPU based mining...or the other niche markets where eventually the cost of custom asic becomes to attractive to ignore even if you as a consume have to handle a few years of growing pains.

> Ultimately GPU's main use is multimedia and graphics focused

This has long ceased to be true, especially for data center focused gpus from the last few years; the "gpu" moniker is really a misnomer / historical artifact.

Post reply on HN