Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

How Taalas “prints” LLM onto a chip?

anuragk.com

Post reply on HN
Print view View original

21–30 of 266 posts

Re: How Taalas “prints” LLM onto a chip?

#21

Post by londons_explore »

generuso wrote: post #4
rustyhancock wrote: post #2

Edit: reading the below it looks like I'm quite wrong here but I've left the comment... The single transistor multiply is intriguing. Id assume they are layers of FMA operating in the log domain. But everything tells me that would be too noisy and error prone to work. On the other hand my mind is completely biased to the digital world. If they stay in the log domain and use a resistor network for multiplication, and…

The document referenced in the blog does not say anything about the single transistor multiply. However, [1] provides the following description: "Taalas’ density is also helped by an innovation which stores a 4-bit model parameter and does multiplication on a single transistor, Bajic said (he declined to give further details but confirmed that compute is still fully digital)." [1] https://www.eetimes.com/taalas-speci…

It'll be different gates on the transistor for the different bits, and you power only one set depending on which bit of the result you wish to calculate.

Some would call it a multi-gate transistor, whilst others would call it multiple transistors in a row...

Re: How Taalas “prints” LLM onto a chip?

#22

Post by moralestapia »

>HOW NVIDIA GPUs process stuff? (Inefficiency 101)

Wow. Massively ignorant take. A modern GPUs is an amazing feat of engineering, particularly about making computation more efficient (low power/high throughput).

Then proceeds to explain, wrongly, how inference is supposssedly implemented and draws conclusions from there ...

Re: How Taalas “prints” LLM onto a chip?

#23

Post by cpldcpu »

abrichr wrote: post #13

ChatGPT Deep Research dug through Taalas' WIPO patent filings and public reporting to piece together a hypothesis. Next Platform notes at least 14 patents filed [1]. The two most relevant: "Large Parameter Set Computation Accelerator Using Memory with Parameter Encoding" [2] "Mask Programmable ROM Using Shared Connections" [3] The "single transistor multiply" could be multiplication by routing, not arithmetic. Patent…

It could simply be bit serial. With 4 bit weights you only need four serial addition steps, which is not an issue if the weight are stored nearby in a rom.

Re: How Taalas “prints” LLM onto a chip?

#24

Post by XorNot »

roncesvalles wrote: post #20
owenpalmer wrote: post #5

> Kinda like a CD-ROM/Game cartridge, or a printed book, it only holds one model and cannot be rewritten. Imagine a slot on your computer where you physically pop out and replace the chip with different models, sort of like a Nintendo DS.

That slot is called USB-C. I can fully imagine inference ASICs coming in powerbank form factor that you'd just plug and play.

Pretty sure it'd just be a thumbdrive. Are the Taalas chips particularly large in surface area?

Re: How Taalas “prints” LLM onto a chip?

#25

Post by cpldcpu »

I wonder how well this works with MoE architectures?

For dense LLMs, like llama-3.1-8B, you profit a lot from having all the weights available close to the actual multiply-accumulate hardware.

With MoE, it is rather like a memory lookup. Instead of a 1:1 pairing of MACs to stored weights, you suddenly are forced to have a large memory block next to a small MAC block. And once this mismatch becomes large enough, there is a huge gain by using a highly optimized memory process for the memory instead of mask ROM.

At that point we are back to a chiplet approach...

Re: How Taalas “prints” LLM onto a chip?

#26

Post by beAroundHere »

moralestapia wrote: post #22

>HOW NVIDIA GPUs process stuff? (Inefficiency 101) Wow. Massively ignorant take. A modern GPUs is an amazing feat of engineering, particularly about making computation more efficient (low power/high throughput). Then proceeds to explain, wrongly, how inference is supposssedly implemented and draws conclusions from there ...

Hey, Can you please point out explain the inaccuracies in the article?

I had written this post to have a higher level understanding of traditional vs Taalas's inference. So it does abstracts lots of things.

Re: How Taalas “prints” LLM onto a chip?

#28

Post by hagbard_c »

londons_explore wrote: post #21
generuso wrote: post #4

Earlier quoted context omitted.

The document referenced in the blog does not say anything about the single transistor multiply. However, [1] provides the following description: "Taalas’ density is also helped by an innovation which stores a 4-bit model parameter and does multiplication on a single transistor, Bajic said (he declined to give further details but confirmed that compute is still fully digital)." [1] https://www.eetimes.com/taalas-speci…

It'll be different gates on the transistor for the different bits, and you power only one set depending on which bit of the result you wish to calculate. Some would call it a multi-gate transistor, whilst others would call it multiple transistors in a row...

That, or a resistor ladder with 4 bit branches connected to a single gate, possibly with a capacitor in between, representing the binary state as an analogue voltage, i.e. an analogue-binary computer. If it works for flash memory it could work for this application as well.

Re: How Taalas “prints” LLM onto a chip?

#29

Post by fercircularbuf »

punnerud wrote: post #27

Could we all get bigger FPGAs and load the model onto it using the same technique?

I thought about this exact question yesterday. Curious to know why we couldn't, if it isn't feasible. Would allow one to upgrade to the next model without fabricating all new hardware.

Re: How Taalas “prints” LLM onto a chip?

#30

Post by thesz »

8B coefficients are packed into 53B transistors, 6.5 transistors per coefficient. Two-inputs NAND gate takes 4 transistors and register takes about the same. One coefficient gets processed (multiplied by and result added to a sum) with less than two two-inputs NAND gates.

I think they used block quantization: one can enumerate all possible blocks for all (sorted) permutations of coefficients and for each layer place only these blocks that are needed there. For 3-bit coefficients and block size of 4 coefficients only 330 different blocks are needed.

Matrices in the llama 3.1 are 4096x4096, 16M coefficients. They can be compressed into only 330 blocks, if we assume that all coefficients' permutations are there, and network of correct permutations of inputs and outputs.

Assuming that blocks are the most area consuming part, we have block's transistor budget of about 250 thousands of transistors, or 30 thousands of 2-inputs NAND gates per block.

250K transistors per block * 330 blocks / 16M transistors = about 5 transistors per coefficient.

Looks very, very doable.

It does look doable even for FP4 - these are 3-bit coefficients in disguise.

Post reply on HN