Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

How Taalas “prints” LLM onto a chip?

anuragk.com

Post reply on HN
Print view View original

151–160 of 266 posts

Re: How Taalas “prints” LLM onto a chip?

#151

Post by ac29 »

qoez wrote: post #103

> It took them two months, to develop chip for Llama 3.1 8B. In the AI world where one week is a year, it's super slow. But in a world of custom chips, this is supposed to be insanely fast. LLama 3.1 is like 2 years at this point. Taking two months to convert a model that only updates every 2 years is very fast

2 months of design work is fast, but how much time does fabrication, packaging, testing add? And that just gets you chips, whatever products incorporate them also need to be built and tested.

Re: How Taalas “prints” LLM onto a chip?

#152

Post by stavros »

grayhatter wrote: post #147
stavros wrote: post #76

Earlier quoted context omitted.

USB-C can do up to 240W. These days I power all my devices with a USB hub, even my Lipo charger.

Have you seen a device that can supply 240w and act as a data host? Or is the 240w only from dedicated chargers?

I haven't seen one, but I also don't tend to use it for anything other than a power supply, so I wouldn't know. Since the standard supports it, though, it's just a matter of the market needing a device like that.

Re: How Taalas “prints” LLM onto a chip?

#153

Post by theptip »

bsenftner wrote: post #93

I'm surprised people are surprised. Of course this is possible, and of course this is the future. This has been demonstrated already: why do you think we even have GPUs at all?! Because we did this exact same transition from running in software to largely running in hardware for all 2D and 3D Computer Graphics. And these LLMs are practically the same math, it's all just obvious and inevitable, if you're paying attent…

Are people surprised?

I think the interesting point is the transition time. When is it ROI-positive to tape out a chip for your new model? There’s a bunch of fun infra to build to make this process cheaper/faster and I imagine MoE will bring some challenges.

Re: How Taalas “prints” LLM onto a chip?

#154

Post by luckydata »

rembal wrote: post #148
bsenftner wrote: post #93

I'm surprised people are surprised. Of course this is possible, and of course this is the future. This has been demonstrated already: why do you think we even have GPUs at all?! Because we did this exact same transition from running in software to largely running in hardware for all 2D and 3D Computer Graphics. And these LLMs are practically the same math, it's all just obvious and inevitable, if you're paying attent…

It's not certain this is the future: the obvious trade off is lack of flexibility: not only when a new model comes out, but also varying demand in the data centers - one day people want more LLM queries, another day more diffusion queries. Aaand, this blocks the holly grail of self improving models, beyond in-context learning. A realistic use case? More efficient vision based drone targeting in Ukraine/Taiwan/ whatev…

It doesn't have be to true for all models to be useful. Thinking about small models running on phones or edge devices deployed in the field that would be a perfect use case for a "printed model".

Re: How Taalas “prints” LLM onto a chip?

#155

Post by saati »

pwarner wrote: post #136
bsenftner wrote: post #93

I'm surprised people are surprised. Of course this is possible, and of course this is the future. This has been demonstrated already: why do you think we even have GPUs at all?! Because we did this exact same transition from running in software to largely running in hardware for all 2D and 3D Computer Graphics. And these LLMs are practically the same math, it's all just obvious and inevitable, if you're paying attent…

I'd be kind of shocked if Nvidia isn't playing with this. I don't expect it's like super commercially viable today, but for sure things need to trend to radically more efficient AI solutions.

These are chips that become e-waste the second a better a model comes out, and nvidia is already limited by TSMC capacity.

Re: How Taalas “prints” LLM onto a chip?

#156

Post by JLO64 »

briansm wrote: post #67

I wonder if you could use the same technique (RAM models as ROM) for something like Whisper Speech-to-text, where the models are much smaller (around a Gigabyte) for a super-efficient single-chip speech recognition solution with tons of context knowledge.

Right now I have to wait 10 minutes at a time for the 2+ hour long transcriptions I've uploaded to Voxstral to process. The speed up here could be immense and worthwhile to so many customers of these products.

Re: How Taalas “prints” LLM onto a chip?

#157

Post by darkwater »

rembal wrote: post #148
bsenftner wrote: post #93

I'm surprised people are surprised. Of course this is possible, and of course this is the future. This has been demonstrated already: why do you think we even have GPUs at all?! Because we did this exact same transition from running in software to largely running in hardware for all 2D and 3D Computer Graphics. And these LLMs are practically the same math, it's all just obvious and inevitable, if you're paying attent…

It's not certain this is the future: the obvious trade off is lack of flexibility: not only when a new model comes out, but also varying demand in the data centers - one day people want more LLM queries, another day more diffusion queries. Aaand, this blocks the holly grail of self improving models, beyond in-context learning. A realistic use case? More efficient vision based drone targeting in Ukraine/Taiwan/ whatev…

In a not-too-distant future (5 years?) small LLMs will be good enough to be used as generic models for most tasks. And if you have a dedicated ASIC small enough to fit in an iPhone, you have a truly local AI device with the bonus point that you get something really new to sell in every new generation (i.e. acces to an even more powerful model)

Re: How Taalas “prints” LLM onto a chip?

#158

Post by ac29 »

NinjaTrance wrote: post #60
lm28469 wrote: post #50

Who's going to pay for custom chips when they shit out new models every two weeks and their deluded CEOs keep promising AGI in two release cycles?

To run Llama 3.1 8B locally, you would need a GPU with a minimum of 16 GB of VRAM, such as an NVIDIA RTX 3090. Talas promises a 10x higher throughtput, being 10x cheaper and using 10x less electricity. Looks like a good value proposition.

> To run Llama 3.1 8B locally, you would need a GPU with a minimum of 16 GB of VRAM, such as an NVIDIA RTX 3090

In full precision, yes. But this talaas chip uses a heavily quantized version (the article calls it "3/6 bit quant", probably similar to Q4_K_M). You dont even need a GPU to run that with reasonable performance, a CPU is fine.

Re: How Taalas “prints” LLM onto a chip?

#160

Post by cpldcpu »

thesz wrote: post #30

8B coefficients are packed into 53B transistors, 6.5 transistors per coefficient. Two-inputs NAND gate takes 4 transistors and register takes about the same. One coefficient gets processed (multiplied by and result added to a sum) with less than two two-inputs NAND gates. I think they used block quantization: one can enumerate all possible blocks for all (sorted) permutations of coefficients and for each layer place…

They mentioned that they using strong quantization (iirc 3bit) and that the model was degradeted from that. Also, they don't have to use transistors to store the bits.
Post reply on HN