Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

How Taalas “prints” LLM onto a chip?

anuragk.com

Post reply on HN
Print view View original

31–40 of 266 posts

Re: How Taalas “prints” LLM onto a chip?

#31

Post by dmurray »

XorNot wrote: post #24
roncesvalles wrote: post #20

Earlier quoted context omitted.

That slot is called USB-C. I can fully imagine inference ASICs coming in powerbank form factor that you'd just plug and play.

Pretty sure it'd just be a thumbdrive. Are the Taalas chips particularly large in surface area?

The only product they've announced at the moment [0] is a PCI-e card. It's more like a small power bank than a big thumb drive.

But sure, the next generation could be much smaller. It doesn't require battery cells, (much) heat management, or ruggedization, all of which put hard limits on how much you can miniaturise power banks.

[0] https://taalas.com/the-path-to-ubiquitous-ai/

Re: How Taalas “prints” LLM onto a chip?

#32

Post by jsjdjrjdjdjrn »

rustyhancock wrote: post #2

Edit: reading the below it looks like I'm quite wrong here but I've left the comment... The single transistor multiply is intriguing. Id assume they are layers of FMA operating in the log domain. But everything tells me that would be too noisy and error prone to work. On the other hand my mind is completely biased to the digital world. If they stay in the log domain and use a resistor network for multiplication, and…

I'd expect this is analog multiplication with voltage levels being ADC'd out for the bits they want. If you think about it, it makes the whole thing very analog.

Re: How Taalas “prints” LLM onto a chip?

#33

Post by jsjdjrjdjdjrn »

jsjdjrjdjdjrn wrote: post #32
rustyhancock wrote: post #2

Edit: reading the below it looks like I'm quite wrong here but I've left the comment... The single transistor multiply is intriguing. Id assume they are layers of FMA operating in the log domain. But everything tells me that would be too noisy and error prone to work. On the other hand my mind is completely biased to the digital world. If they stay in the log domain and use a resistor network for multiplication, and…

I'd expect this is analog multiplication with voltage levels being ADC'd out for the bits they want. If you think about it, it makes the whole thing very analog.

Note: reading further down, my speculation is wrong.

Re: How Taalas “prints” LLM onto a chip?

#34

Post by thesz »

XorNot wrote: post #24
roncesvalles wrote: post #20

Earlier quoted context omitted.

That slot is called USB-C. I can fully imagine inference ASICs coming in powerbank form factor that you'd just plug and play.

Pretty sure it'd just be a thumbdrive. Are the Taalas chips particularly large in surface area?

800 mm2, about 90mm per side, if imagined as a square. Also, 250 W of power consumption.

The form factor should be anything but thumbdrive.

Re: How Taalas “prints” LLM onto a chip?

#35

Post by pests »

cpldcpu wrote: post #25

I wonder how well this works with MoE architectures? For dense LLMs, like llama-3.1-8B, you profit a lot from having all the weights available close to the actual multiply-accumulate hardware. With MoE, it is rather like a memory lookup. Instead of a 1:1 pairing of MACs to stored weights, you suddenly are forced to have a large memory block next to a small MAC block. And once this mismatch becomes large enough, there…

For comparison I wanted to write on how Google handles MoE archs with its TPUv4 arch.

They use Optical Circuit Switches, operating via MEMS mirrors, to create highly reconfigurable, high-bandwidth 3D torus topologies. The OCS fabric allows 4,096 chips to be connected in a single pod, with the ability to dynamically rewire the cluster to match the communication patterns of specific MoE models.

The 3D torus connects 64-chip cubes with 6 neighbors each. TPUv4 also contains 2 SparseCores which specialize handling high-bandwidth, non-contiguous memory accesses.

Of course this is a DC level system, not something on a chip for your pc, but just want to express the scale here.

*ed: SpareCubes to SparseCubes

Re: How Taalas “prints” LLM onto a chip?

#36

Post by generuso »

abrichr wrote: post #13

ChatGPT Deep Research dug through Taalas' WIPO patent filings and public reporting to piece together a hypothesis. Next Platform notes at least 14 patents filed [1]. The two most relevant: "Large Parameter Set Computation Accelerator Using Memory with Parameter Encoding" [2] "Mask Programmable ROM Using Shared Connections" [3] The "single transistor multiply" could be multiplication by routing, not arithmetic. Patent…

LSI Logic and VLSI Systems used to do such things in 1980s -- they produced a quantity of "universal" base chips, and then relatively inexpensively and quickly customized them for different uses and customers, by adding a few interconnect layers on top. Like hardwired FPGAs. Such semi-custom ASICs were much less expensive than full custom designs, and one could order them in relatively small lots.

Taalas of course builds base chips that are already closely tailored for a particular type of models. They aim to generate the final chips with the model weights baked into ROMs in two months after the weights become available. They hope that the hardware will be profitable for at least some customers, even if the model is only good enough for a year. Assuming they do get superior speed and energy efficiency, this may be a good idea.

Re: How Taalas “prints” LLM onto a chip?

#37

Post by pfortuny »

thesz wrote: post #34
XorNot wrote: post #24

Earlier quoted context omitted.

Pretty sure it'd just be a thumbdrive. Are the Taalas chips particularly large in surface area?

800 mm2, about 90mm per side, if imagined as a square. Also, 250 W of power consumption. The form factor should be anything but thumbdrive.

mmmhhhhh 800mm2 ~= (30mm)2, which is more like a (biggish) thumb drive.

Re: How Taalas “prints” LLM onto a chip?

#39

Post by thesz »

pfortuny wrote: post #37
thesz wrote: post #34

Earlier quoted context omitted.

800 mm2, about 90mm per side, if imagined as a square. Also, 250 W of power consumption. The form factor should be anything but thumbdrive.

mmmhhhhh 800mm2 ~= (30mm)2, which is more like a (biggish) thumb drive.

Thanks!

I haven't had my coffee yet. ;)

Re: How Taalas “prints” LLM onto a chip?

#40

Post by wmf »

londons_explore wrote: post #18

So why only 30,000 tokens per second? If the chip is designed as the article says, they should be able to do 1 token per clock cycle... And whilst I'm sure the propagation time is long through all that logic, it should still be able to do tens of millions of tokens per second...

You still need to do a forward pass per token. With massive batching and full pipelining you might be able to break the dependencies and output one token per cycle but clearly they aren't doing that.
Post reply on HN