Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

How Taalas “prints” LLM onto a chip?

anuragk.com

Post reply on HN
Print view View original

221–230 of 266 posts

Re: How Taalas “prints” LLM onto a chip?

#221

Post by theptip »

byran wrote: post #3

This would be a very interesting future. I can imagine Gemma 5 Mini running locally on hardware, or a hard-coded "AI core" like an ALU or media processor that supports particular encoding mechanisms like H.264, AV1, etc. Other than the obvious costs (but Taalas seems to be bringing back the structured ASIC era so costs shouldn't be that low [1]), I'm curious why this isn't getting much attention from larger companies…

> I'm curious why this isn't getting much attention from larger companies

I would be shocked if Google isn’t working on this right now. They build their own TPUs, this is an extremely obvious direction from there.

(And there are plenty of interesting co-design questions that only the frontier labs can dabble with; Taalas is stuck working around architectural quirks like “top-8 MoE”, Google can just rework the architecture hyperparameters to whatever gets best results in silico.)

Re: How Taalas “prints” LLM onto a chip?

#222

Post by 8note »

throwup238 wrote: post #218
sowbug wrote: post #182

Earlier quoted context omitted.

Ugh, quick, everyone start panic-buying FPGAs now.

largest FPGAs have on the order of tens of millions of logic cells/elements. They’re not even remotely big enough to emulate these designs except to validate small parts of it at a time and unlike memory chips or GPUs, companies don’t need millions of them to scale infrastructure. (The chips also cost tens of thousands of dollars each)

they also arent power friendly

Re: How Taalas “prints” LLM onto a chip?

#223

Post by mirekrusin »

cpldcpu wrote: post #160
thesz wrote: post #30

8B coefficients are packed into 53B transistors, 6.5 transistors per coefficient. Two-inputs NAND gate takes 4 transistors and register takes about the same. One coefficient gets processed (multiplied by and result added to a sum) with less than two two-inputs NAND gates. I think they used block quantization: one can enumerate all possible blocks for all (sorted) permutations of coefficients and for each layer place…

They mentioned that they using strong quantization (iirc 3bit) and that the model was degradeted from that. Also, they don't have to use transistors to store the bits.

gpt-oss is fp4 - they're saying they'll next try mid size one, I'm guessing gpt-oss-20b then large one, i'm guessing gpt-oss-120b as their hardware is fp4 friendly

Re: How Taalas “prints” LLM onto a chip?

#224

Post by fho »

kop316 wrote: post #194

Ohh neat! A generalized version of this was the topic of my PhD dissertation: https://kilthub.cmu.edu/articles/thesis/Modern_Gate_Array_De... And they are likely doing something similar to put their LLMs in silicon. I would believe a 10x electricity boost along with it being much faster. The idea is that you can create a sea of generalized standard cells and it makes for a gate array at the manufacturing layer. This…

I think their "4-bit multiplier with a single transistor" bit is hinting at them using transistors in the sun-threshold regime.

Re: How Taalas “prints” LLM onto a chip?

#225

Post by ivan_gammel »

bigyabai wrote: post #201
ivan_gammel wrote: post #99

Earlier quoted context omitted.

When output is good enough, other considerations become more important. Most people on this planet cannot afford even an AI subscription, and cost of tokens is prohibitive to many low margin businesses. Privacy and personalization matter too, data sovereignty is a hot topic. Besides, we already see how focus has shifted to orchestration, which can be done on CPU and is cheap - software optimizations may compensate ha…

This is the same justification that was used to ship the (now almost entirely defunct) NPUs on Apple and Android devices alike. The A18 iPhone chip has 15b transistors for the GPU and CPU; the Taalas ASIC has 53b transistors dedicated to inference alone. If it's anything like NPUs, almost all vendors will bypass the baked-in silicon to use GPU acceleration past a certain point. It makes much more sense to ship a CUDA…

Why are you thinking about phones specifically? Most heavy users are on laptops and workstations. On smartphones there might be a few more innovations necessary (low latency AI computing on the edge?)

Re: How Taalas “prints” LLM onto a chip?

#226

Post by ivan_gammel »

ACCount37 wrote: post #205
ivan_gammel wrote: post #190

Earlier quoted context omitted.

The generation of frontier models from H1 2025 is the good enough benchmark.

Flash forward one year and it'll be H1 2026.

I don’t see why. Today frontier models are already 2 generations ahead of good enough. For many users they did not offer substantial improvement, sometimes things got even worse. What is going to happen within 1 year that will make users desire something beyond already working solution? LLMs are reaching maturity faster than smartphones, which now are good enough to stay on the same model for at least 5-6 years.

Re: How Taalas “prints” LLM onto a chip?

#227

Post by yunohn »

fxnn wrote: post #217
yunohn wrote: post #181

Earlier quoted context omitted.

But the BTC mining algorithm has not and will not change. That’s the only reason ASICs atleast make a bit of sense for crypto. AI being static weights is already challenged with the frequent model updates we already see - but may even be a relic once we find a new architecture.

We can expect the model landscape to consolidate some day. Progress will become slower, innovations will become smaller. Not tomorrow, not next year, but the time will come. And then it'll increasingly make sense to build such a chip into laptops, smartphones, wearables. Not for high-end tasks, but to drive the everyday bread-and-butter tasks.

The world continues to evolve, in a way that requires flexibility - not more constraints. I just fail to see a future where we want less general purpose computers, and more hard-wired ones? Would be interesting to be proven wrong though!

Re: How Taalas “prints” LLM onto a chip?

#228

Post by wmf »

slow_typist wrote: post #220
wmf wrote: post #207

Earlier quoted context omitted.

The Taalas approach is much more expensive than the NPU that phones already have.

Yes but not in five years. The chips will be dirt cheap by then. We‘ll get “intelligent” washing machines that will discuss the amount of detergent and eventually berate us. Toasters with voice input. And really annoying elevators. Also bugs that keep an extremely low RF profile (only phoning home when the target is talking business).

No, Taalas requires more silicon which will always cost more than storing weights in DRAM.

Re: How Taalas “prints” LLM onto a chip?

#229

Post by yunohn »

fxnn wrote: post #215
yunohn wrote: post #213

Earlier quoted context omitted.

IME llama et all require LoRA or fine-tuning to be usable. That's their real value vs closed source massive models, and their small size makes this possible, appealing, and doable on a recurring basis as things evolve. Again, rendering ASICs useless.

Read the blog post. It mentions that their chip has a small SRAM which can store LoRA.

Neither the blog nor Taalas' original post specify what speed to expect when using the SRAM in conjunction with the baked-in weights? To be taken seriously, that is really necessary to explain in detail, than a passing mention.

Re: How Taalas “prints” LLM onto a chip?

#230

Post by kop316 »

fho wrote: post #224
kop316 wrote: post #194

Ohh neat! A generalized version of this was the topic of my PhD dissertation: https://kilthub.cmu.edu/articles/thesis/Modern_Gate_Array_De... And they are likely doing something similar to put their LLMs in silicon. I would believe a 10x electricity boost along with it being much faster. The idea is that you can create a sea of generalized standard cells and it makes for a gate array at the manufacturing layer. This…

I think their "4-bit multiplier with a single transistor" bit is hinting at them using transistors in the sun-threshold regime.

So something that you can do with PDKs is add your own custom standard cell and tell the EDA tools to use them. This is actually pretty smart, this way you can use most of the foundry cells (which have been extensively validated) and focus on things like this "magic multiplier", that you will have to manually validate. This also makes porting across tech nodes easier if you manage only a handful of custom cells versus a completely custom design.

(I have my guesses as to what that is, but I admittedly don't know enough about that particular part of the field to give anything but a guess).

Post reply on HN