[flagged]
How Taalas “prints” LLM onto a chip?
41–50 of 266 posts
Re: How Taalas “prints” LLM onto a chip?
#42Could we all get bigger FPGAs and load the model onto it using the same technique?
You could [1], but it is not very cheap -- the 32GB development board with the FPGA used in the article used to cost about $16K.
Re: How Taalas “prints” LLM onto a chip?
#43>HOW NVIDIA GPUs process stuff? (Inefficiency 101) Wow. Massively ignorant take. A modern GPUs is an amazing feat of engineering, particularly about making computation more efficient (low power/high throughput). Then proceeds to explain, wrongly, how inference is supposssedly implemented and draws conclusions from there ...
Arguably DRAM-based GPUs/TPUs are quite inefficient for inference compared to SRAM-based Groq/Cerebras. GPUs are highly optimized but they still lose to different architectures that are better suited for inference.
Re: How Taalas “prints” LLM onto a chip?
#44> Kinda like a CD-ROM/Game cartridge, or a printed book, it only holds one model and cannot be rewritten. Imagine a slot on your computer where you physically pop out and replace the chip with different models, sort of like a Nintendo DS.
A cartridge slot for models is a fun idea. Instead of one chip running any model, you get one model or maybe a family of models per chip at (I assume) much better perf/watt. Curious whether the economics work out for consumer use or if this stays in the embedded/edge space.
Plug it into skull bone. Neuralink + slot for a model that you can buy in s grocery store instead of prepaid Netflix card.
Re: How Taalas “prints” LLM onto a chip?
#45Could we all get bigger FPGAs and load the model onto it using the same technique?
FPGAs have really low density so that would be ridiculously inefficient, probably requiring ~100 FPGAs to load the model. You'd be better off with Groq.
Not sure what you're on but I think what you said is incorrect. You can use hi-density HBM-enabled FPGA with (LP)DDR5 with sufficient number of logic elements to implement the inference. Reason why we don't see it in action is most likely in the fact that such FPGAs are insanely expensive and not so available off-the-shelf as the GPUs are.
Re: How Taalas “prints” LLM onto a chip?
#46So why only 30,000 tokens per second? If the chip is designed as the article says, they should be able to do 1 token per clock cycle... And whilst I'm sure the propagation time is long through all that logic, it should still be able to do tens of millions of tokens per second...
Reading from and to memory alone takes much more than a clock cycle.
Re: How Taalas “prints” LLM onto a chip?
#47> Kinda like a CD-ROM/Game cartridge, or a printed book, it only holds one model and cannot be rewritten. Imagine a slot on your computer where you physically pop out and replace the chip with different models, sort of like a Nintendo DS.
That slot is called USB-C. I can fully imagine inference ASICs coming in powerbank form factor that you'd just plug and play.
This would be a hell of a hot power bank. It uses about as much power as my oven. So probably more like inside a huge cooling device outside the house. Or integrated into the heating system of the house.
(Still compelling!)
Re: How Taalas “prints” LLM onto a chip?
#48This would be a very interesting future. I can imagine Gemma 5 Mini running locally on hardware, or a hard-coded "AI core" like an ALU or media processor that supports particular encoding mechanisms like H.264, AV1, etc. Other than the obvious costs (but Taalas seems to be bringing back the structured ASIC era so costs shouldn't be that low [1]), I'm curious why this isn't getting much attention from larger companies…
Well even programmable ASICs like Cerebras and Groq give many-multiples speedup over GPUs and the market has hardly reacted at all.
> market has hardly reacted at all
Guess who acqui-hired Groq to push this into GPUs?
The name GPU has been an anachronism for a couple of years now.
Re: How Taalas “prints” LLM onto a chip?
#49> Kinda like a CD-ROM/Game cartridge, or a printed book, it only holds one model and cannot be rewritten. Imagine a slot on your computer where you physically pop out and replace the chip with different models, sort of like a Nintendo DS.
Would somewhat work except for the power usage.
I doubt it would scale linearly, but for home use 170 tokens/s at 2.5W would be cool; 17 tokens/s at 0,25W would be awesome.
On the other hand, this may be a step towards positronic brains (https://en.wikipedia.org/wiki/Positronic_brain)
Re: How Taalas “prints” LLM onto a chip?
#50Who's going to pay for custom chips when they shit out new models every two weeks and their deluded CEOs keep promising AGI in two release cycles?