How Taalas “prints” LLM onto a chip? • Hacker News Board

Re: How Taalas “prints” LLM onto a chip?

#201

Post by bigyabai » Sun, Feb 22, 2026, 6:32 PM UTC

ivan_gammel wrote: post #99

arisAlexis wrote: post #90

Earlier quoted context omitted.

and run an outdated model for 3 years while progress is exponential? what is the point of that

When output is good enough, other considerations become more important. Most people on this planet cannot afford even an AI subscription, and cost of tokens is prohibitive to many low margin businesses. Privacy and personalization matter too, data sovereignty is a hot topic. Besides, we already see how focus has shifted to orchestration, which can be done on CPU and is cheap - software optimizations may compensate ha…

This is the same justification that was used to ship the (now almost entirely defunct) NPUs on Apple and Android devices alike.

The A18 iPhone chip has 15b transistors for the GPU and CPU; the Taalas ASIC has 53b transistors dedicated to inference alone. If it's anything like NPUs, almost all vendors will bypass the baked-in silicon to use GPU acceleration past a certain point. It makes much more sense to ship a CUDA-style flexible GPGPU architecture.

Re: How Taalas “prints” LLM onto a chip?

#202

Post by cedws » Sun, Feb 22, 2026, 6:33 PM UTC

Twirrim wrote: post #145

cedws wrote: post #119

Earlier quoted context omitted.

The network latency bit deserves more attention. I’ve been trying to find out where AI companies are physically serving LLMs from but it’s difficult to find information about this. If I’m sitting in London and use Claude, where are the requests actually being served? The ideal world would be an edge network like Cloudflare for LLMs so a nearby POP serves your requests. I’m not sure how viable this is. On classic hard…

> The network latency bit deserves more attention. I’ve been trying to find out where AI companies are physically serving LLMs from but it’s difficult to find information about this. If I’m sitting in London and use Claude, where are the requests actually being served? Unfortunately, as with most of the AI providers, it's wherever they've been able to find available power and capacity. They've contracts with all of t…

>You could be having your requests processed halfway around the world, or right next door, from one minute to the next

Wow, any source for this? It would explain why they vary between feeling really responsive and really delayed.

Re: How Taalas “prints” LLM onto a chip?

#203

Post by bigyabai » Sun, Feb 22, 2026, 6:34 PM UTC

hamdingers wrote: post #198

bigyabai wrote: post #195

Earlier quoted context omitted.

If you pay $1,500 for a Mistral ASIC that is beaten by a $15 Qwen ASIC that comes out six months later, you'd be feeling pretty dang ridiculous.

I'm equally capable of making up numbers to support my perspective but I don't see the point.

The point is that the GP's mindset is not very ridiculous if you value things by a price/utility ratio. Software and hardware advancements will lead to buyer's remorse faster than people get an ROI from local inference.

Re: How Taalas “prints” LLM onto a chip?

#204

Post by amelius » Sun, Feb 22, 2026, 6:41 PM UTC

cpldcpu wrote: post #160

thesz wrote: post #30

8B coefficients are packed into 53B transistors, 6.5 transistors per coefficient. Two-inputs NAND gate takes 4 transistors and register takes about the same. One coefficient gets processed (multiplied by and result added to a sum) with less than two two-inputs NAND gates. I think they used block quantization: one can enumerate all possible blocks for all (sorted) permutations of coefficients and for each layer place…

They mentioned that they using strong quantization (iirc 3bit) and that the model was degradeted from that. Also, they don't have to use transistors to store the bits.

I think they are talking about the transistors that apply the weights to the inputs.

Re: How Taalas “prints” LLM onto a chip?

#205

Post by ACCount37 » Sun, Feb 22, 2026, 6:46 PM UTC

ivan_gammel wrote: post #190

ACCount37 wrote: post #173

Earlier quoted context omitted.

"Good enough" open weights models were "almost there" since 2022. I distrust the notion. The bar of "good enough" seems to be bolted to "like today's frontier models", and frontier model performance only ever goes up.

The generation of frontier models from H1 2025 is the good enough benchmark.

Flash forward one year and it'll be H1 2026.

Re: How Taalas “prints” LLM onto a chip?

#206

Post by wmf » Sun, Feb 22, 2026, 6:50 PM UTC

menaerus wrote: post #45

wmf wrote: post #38

Earlier quoted context omitted.

FPGAs have really low density so that would be ridiculously inefficient, probably requiring ~100 FPGAs to load the model. You'd be better off with Groq.

Not sure what you're on but I think what you said is incorrect. You can use hi-density HBM-enabled FPGA with (LP)DDR5 with sufficient number of logic elements to implement the inference. Reason why we don't see it in action is most likely in the fact that such FPGAs are insanely expensive and not so available off-the-shelf as the GPUs are.

Yeah, FPGA+HBM works but it has no advantage over GPU+HBM. If you want to store weights in FPGA LUTs/SRAM for insane speed you're going to need a lot of FPGAs because each one has very little capacity.

Re: How Taalas “prints” LLM onto a chip?

#207

Post by wmf » Sun, Feb 22, 2026, 6:58 PM UTC

darkwater wrote: post #157

rembal wrote: post #148

Earlier quoted context omitted.

It's not certain this is the future: the obvious trade off is lack of flexibility: not only when a new model comes out, but also varying demand in the data centers - one day people want more LLM queries, another day more diffusion queries. Aaand, this blocks the holly grail of self improving models, beyond in-context learning. A realistic use case? More efficient vision based drone targeting in Ukraine/Taiwan/ whatev…

In a not-too-distant future (5 years?) small LLMs will be good enough to be used as generic models for most tasks. And if you have a dedicated ASIC small enough to fit in an iPhone, you have a truly local AI device with the bonus point that you get something really new to sell in every new generation (i.e. acces to an even more powerful model)

The Taalas approach is much more expensive than the NPU that phones already have.

Re: How Taalas “prints” LLM onto a chip?

#208

Post by dangus » Sun, Feb 22, 2026, 7:00 PM UTC

yunohn wrote: post #181

RobotToaster wrote: post #118

Earlier quoted context omitted.

It reminds me of the switch from GPUs to ASICs in bitcoin mining. I've been expecting this to happen.

But the BTC mining algorithm has not and will not change. That’s the only reason ASICs atleast make a bit of sense for crypto. AI being static weights is already challenged with the frequent model updates we already see - but may even be a relic once we find a new architecture.

Sounds to me like there’s potential to use these for established models to provide cost/scale advantage while frontier models will run in the existing setup.

Re: How Taalas “prints” LLM onto a chip?

#209

Post by wmf » Sun, Feb 22, 2026, 7:00 PM UTC

sowbug wrote: post #188

MarcLore wrote: post #78

[dead]

As scary as it sounds today, a lightning-quick zero latency non-networked local LLM could provide value in an application like a self-driving car. It would be a level below Waymo's remote human support, so if the car couldn't figure out how to deal with a weird situation, it could ask the LLM what to do, hopefully avoiding the need to phone home (and perhaps handling cases where it couldn't phone home).

Waymo already has on-board NPU(s) with Transformer model(s) that are cheaper than Taalas.

Re: How Taalas “prints” LLM onto a chip?

#210

Post by wmf » Sun, Feb 22, 2026, 7:04 PM UTC

adeelk93 wrote: post #150

Aurornis wrote: post #137

Earlier quoted context omitted.

The die size is huge. This isn’t the kind of chip that would go into your MacBook, let alone an iPhone. It’s for cloud based servers.

And computers used to be the size of a room. I think they can get it to iPhone size in the future, this is an early prototype.

That's the part that people are missing: it won't get smaller. It already required heroic optimization to get 8B on one megachip. Taalas is more expensive but faster. It is cheaper per token when running 24x7 but not cheap to buy. It will never be small and never be cheap.