Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

AMD acquires Taalas to boost inference performance by etching models in silicon

theregister.com

Post reply on HN
Print view View original

251–260 of 712 posts

Re: AMD acquires Taalas to boost inference performance by etching models in silicon

#251

Post by QuiDortDine »

teaearlgraycold wrote: post #139
adgjlsfhk1 wrote: post #110

Earlier quoted context omitted.

I don't think this works out from a cost/silicon perspective. Small models already run pretty well in software (since the weights fit in cache) and big models require silicon area proportional to the size of weights. On a mobile device putting a chip like this is competing directly in BOM and power against a whole lot more l3 cache, and the l3 cache makes everything faster

My question is what changes about LLM use cases when you’re getting 1000 tok/s? Models in silicon might dramatically change how we think about them.

Did you use chatjimmy? It's somewhat terrifying to use when you think of the potential results with a better model.

Ok, real life example: I now spend most of my time, as a developer, waiting for the agent to do its thing (after careful prompting, I'm also thinking about work stuff, don't worry I'm not useless). What if it gave back the same excellent results, but instantaneously? Why, then, I certainly would become the bottleneck. So, quite possibly, my last work task would be to plug this agent directly into the ticket system where the domain experts input their feature requests. Maybe we still need 1 developer out of 100, to coordinate releases and all that (ok, say 1 out of 10).

But that's not taking things far enough: why do we need these domain experts at all? Our pitch is clear, and all software-enabled, though it took years to develop. We can just have the clients express their concerns to the AI, directly or indirectly. Have multiple lighting-fast agents with different roles (refactoring agent, new features agent, debugger agent, domain expert agent, etc.). So we fire everyone, maybe keep 1 product owner / devops to keep the trolls out. The cost is still probably 100 times less than it used to be (beyond the initial cost of acquisition of the magic machine or whatever).

But one of these clients, surely, will realize that these 10 years of manual and slowly-automated development can now be emulated in very, very little time. Why not just, say, take screenshots of the entire app and feed them into the magic machine? Why, this way, they could have the service for a tenth of the yearly cost, forever!

And then the economy implodes.

I'm not saying it's THE most likely version of things, I'm saying that at a certain level, quantity (or rather, speed) is a quality all its own. And this new quality might change the world. Let's hope it's for the better!

Re: AMD acquires Taalas to boost inference performance by etching models in silicon

#252

Post by XCSme »

m463 wrote: post #229
XCSme wrote: post #185

Earlier quoted context omitted.

I think this would make sense for consumer hardware, not for AI companies. AI companies constantly update/change stuff, new models come out, new requirements, etc. But if you ship an "ai-powered" dishwasher, it can come with the chip built-in to do computer vision and precisely target each spot, and will be sold as-is with no updates.

subscription "ai-powered" dishwasher with personalized user ads, most of the chip dedicated to "personalized" not spots.

So local personalized ads?

Not sure if that's better or worse than online personalizaed ads...

Re: AMD acquires Taalas to boost inference performance by etching models in silicon

#253

Post by andix »

It would be quite ironic if this technology would render all those AI data centers practically useless. If the next step are just a much smaller amount of expensive chips, and the bottleneck becomes manufacturing those chips fast. Not building huge data centers and fighting for electrical power.

Re: AMD acquires Taalas to boost inference performance by etching models in silicon

#254

Post by vatsachak »

subroutine wrote: post #228
zxspectrum1982 wrote: post #150

Earlier quoted context omitted.

I'd gladly pay for a Claude Opus 4.6 Thinking High in silicon and use it for 1-2 years. It's good enough for many coding tasks.

But Claude Opus 4.6 is not really practical. Taalas' process seems targeted for edge models. Their proof of concept model, for example, is a heavily quantized version of Llama 3.1 8B and even then they acknowledge their custom 3-bit/6-bit representation causes model quality degradation. Taalas is going to have a tough time putting a trillion-parameter model on one conventional die. Their HC1 die is already near the m…

Yeah. But this kinda feels like a bandaid.

Eventually someone will have to solve compute in memory at scale.

Re: AMD acquires Taalas to boost inference performance by etching models in silicon

#255

Post by throwaway173738 »

XCSme wrote: post #185
yumraj wrote: post #121

Given the fast churn of the models, how does it work out? Won’t the silicon etched model already be 1 or more versions behind by the time the silicon comes out. Though if it’s cheap enough, there certainly can be a market for cheaper model inferences.

I think this would make sense for consumer hardware, not for AI companies. AI companies constantly update/change stuff, new models come out, new requirements, etc. But if you ship an "ai-powered" dishwasher, it can come with the chip built-in to do computer vision and precisely target each spot, and will be sold as-is with no updates.

You don’t need this chip to do that. Computer vision has used machine learning for decades. The task you’re describing is pretty rudimentary and an off the shelf model with a control system would do it way cheaper.

Re: AMD acquires Taalas to boost inference performance by etching models in silicon

#256

Post by yumraj »

deadbabe wrote: post #182
yumraj wrote: post #121

Given the fast churn of the models, how does it work out? Won’t the silicon etched model already be 1 or more versions behind by the time the silicon comes out. Though if it’s cheap enough, there certainly can be a market for cheaper model inferences.

You could take your silicon chip and have it re-etched only with model diffs for an upgraded version.

How does that work, as in re-etching of silicon? Any pointers to read?

Re: AMD acquires Taalas to boost inference performance by etching models in silicon

#257

Post by chorizo »

Melatonic wrote: post #134
anthonypasq wrote: post #74

Earlier quoted context omitted.

Personally I think Apple should have acquired them. if you could burn a gemma4 class model into an iphone and actually get extremely low latency and low battery usage it would feel like the future IMO. even if it means you wont get frontier intelligence, there might actually be incentive to buy a new mobile device every year again.

The Taalas chips are not physically small. And part of their secret (if you look at the design) is just locating a bunch of memory soldered on the edges ( I belive higher amounts of SRAM ? )

Baking the base models on to ROM makes a lot of economic sense. SRAM for the KV cache & fine-tunes, not so much. Sure you’d get incredible speeds but it’s not scalable from a die-size or cost perspective.

Rather base model on ROM + KV cache on DRAM is much more scalable. Also this would work great for edge devices that have a 2-5 year lifecycle.

Re: AMD acquires Taalas to boost inference performance by etching models in silicon

#258

Post by yumraj »

XCSme wrote: post #185
yumraj wrote: post #121

Given the fast churn of the models, how does it work out? Won’t the silicon etched model already be 1 or more versions behind by the time the silicon comes out. Though if it’s cheap enough, there certainly can be a market for cheaper model inferences.

I think this would make sense for consumer hardware, not for AI companies. AI companies constantly update/change stuff, new models come out, new requirements, etc. But if you ship an "ai-powered" dishwasher, it can come with the chip built-in to do computer vision and precisely target each spot, and will be sold as-is with no updates.

Makes sense. Actually to expand, I believe this can make a lot of sense for industrial robots and such which have a more or less fixed job and latency matters more, so a well tested model may be more valuable than need to keep updating them

Re: AMD acquires Taalas to boost inference performance by etching models in silicon

#259

Post by XCSme »

throwaway173738 wrote: post #255
XCSme wrote: post #185

Earlier quoted context omitted.

I think this would make sense for consumer hardware, not for AI companies. AI companies constantly update/change stuff, new models come out, new requirements, etc. But if you ship an "ai-powered" dishwasher, it can come with the chip built-in to do computer vision and precisely target each spot, and will be sold as-is with no updates.

You don’t need this chip to do that. Computer vision has used machine learning for decades. The task you’re describing is pretty rudimentary and an off the shelf model with a control system would do it way cheaper.

It was just a random example, you could think of it as being a lot more complex (detect which type of food it is, what detergent to use, how much water, remember patterns, learn over time, adapt, etc.)

Re: AMD acquires Taalas to boost inference performance by etching models in silicon

#260

Post by UncleOxidant »

LarsDu88 wrote: post #44

I'm surprised neither OpenAI nor Anthropic made this move first. The Chinese open weight models are pulling ahead and commoditizing their value proposition. Baking models onto silicon would've been the next logical move to get a moat. Google is already doing this and has an experimental project on top of already having TPUs and cramming their quantized flash onto individual TPUs for inference.

I guess I'm not understanding why this makes sense for AMD to buy Taalas unless they plan to get into hosting. It doesn't seem like a great fit.
Post reply on HN