Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

AMD acquires Taalas to boost inference performance by etching models in silicon

theregister.com

Post reply on HN
Print view View original

411–420 of 712 posts

Re: AMD acquires Taalas to boost inference performance by etching models in silicon

#411

Post by Maxion »

kzrdude wrote: post #390
AussieWog93 wrote: post #382

Earlier quoted context omitted.

Could we not just make bigger wafers, if the technology called for it?

The investment in bigger machines at the fab might set you back billions. I don't know about the lithography technology either, how easy you can scale it to larger wafers?

Lithography is AFAIK the most expensive part. If you wanna scale to larger wafers I'd guess you're talking hundreds of billions and at least a decade. And even then, what will your yield be? Surface area grows to the square of the diameter of a circle, and so will errors. The larger your wafer goes the lower your yield will go, especially if you're going to use the whole wafer in one final product.

Re: AMD acquires Taalas to boost inference performance by etching models in silicon

#413

Post by Maxion »

craftkiller wrote: post #230
yumraj wrote: post #121

Given the fast churn of the models, how does it work out? Won’t the silicon etched model already be 1 or more versions behind by the time the silicon comes out. Though if it’s cheap enough, there certainly can be a market for cheaper model inferences.

I think the real value here is not as a customer-facing agent/chatbot but for for automated processes. Think of all the companies out there that have LLMs doing simple tasks like categorizing customer feedback emails. For such tasks, you don't gain much from better models, so if you could run it 10x cheaper on a slightly older model, it would absolutely be worth it. Pretty much any place people are currently running…

Or as subagents reading code or other text and summarizing them.

Re: AMD acquires Taalas to boost inference performance by etching models in silicon

#414

Post by lelanthran »

adrianN wrote: post #332
chorizo wrote: post #257

Earlier quoted context omitted.

Baking the base models on to ROM makes a lot of economic sense. SRAM for the KV cache & fine-tunes, not so much. Sure you’d get incredible speeds but it’s not scalable from a die-size or cost perspective. Rather base model on ROM + KV cache on DRAM is much more scalable. Also this would work great for edge devices that have a 2-5 year lifecycle.

It is my understanding that just baking the model itself into silicon only gives moderate gains because memory bandwidth remains a bottleneck.

Did you try using the the talaas chat? Something stupid like 18k tokens/second.

Think it's called Askjimmy or similar.

Re: AMD acquires Taalas to boost inference performance by etching models in silicon

#415

Post by selcuka »

aenis wrote: post #400
selcuka wrote: post #270

Earlier quoted context omitted.

Their PoC chips are big, but then it's ridiculously fast (have you seen chatjimmy.ai?). Also they must be holding a bunch of patents.

Its a cool demo, but its gpt-3.5 level stupid, or worse. edit: Ok, I will self-apologize. Its apparently a 3B model. Mighty impressive for what it does.

It's a quantised 8B model (Llama 3.1 8B to be exact).

[1] https://taalas.com/the-path-to-ubiquitous-ai/

Re: AMD acquires Taalas to boost inference performance by etching models in silicon

#416

Post by hugmynutus »

HN is rightly pointing out putting a model into an ASIC is kind of dumb.

HN is failing to understand that AMD knows this well.

Taalas has WO2025217724A1 pending and AMD wants that because it is immediately a function block they can sell to anyone doing FP math, since large (mostly) read only memory banks are ideally suited for that micro-code type stuff.

Re: AMD acquires Taalas to boost inference performance by etching models in silicon

#417

Post by techpression »

kzrdude wrote: post #390
AussieWog93 wrote: post #382

Earlier quoted context omitted.

Could we not just make bigger wafers, if the technology called for it?

The investment in bigger machines at the fab might set you back billions. I don't know about the lithography technology either, how easy you can scale it to larger wafers?

You also need to worry about yields, Apple, AMD etc can sell ”bad” chips as lower core versions, if you’re depending on whole wafer you have little room for error.

Re: AMD acquires Taalas to boost inference performance by etching models in silicon

#419

Post by fsloth »

onion2k wrote: post #387
chorizo wrote: post #257

Earlier quoted context omitted.

Baking the base models on to ROM makes a lot of economic sense. SRAM for the KV cache & fine-tunes, not so much. Sure you’d get incredible speeds but it’s not scalable from a die-size or cost perspective. Rather base model on ROM + KV cache on DRAM is much more scalable. Also this would work great for edge devices that have a 2-5 year lifecycle.

Baking the base models on to ROM makes a lot of economic sense. Less so for consumers though, because it'd mean the phone is out of date in 3 months when a better model comes along.

I don’t think average user _needs_ to solve frontier challenges. ”Call to Jane”, ”turn on the lights” and ”what’s the weather this afternoon” is more like it I would guess.

Ofc if the model has some critical bugs that’s another matter.

Re: AMD acquires Taalas to boost inference performance by etching models in silicon

#420

Post by fsloth »

dyzone wrote: post #366
linzhangrun wrote: post #269

Thinking that five or six years from now, Fable-level intelligence could be provided at 100x the current speed... makes me feel lost. I cannot imagine what the future will look like.

It tells me that they have some kind of insider knowledge that the models have hit their limits and won't be getting much better, and it makes sense economically speaking to just bake the current models and use them for the next 5-10 years. Looks like we're near the top of the S curve.

I’m not sure we are at the top. Swarmed lower tier models appear to scale up in competence. So something like a Fable swarm running at 100x the speed might provide surprises.

But ”top of the curve reached” feels like the likelier scenario.

Post reply on HN