Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

AMD acquires Taalas to boost inference performance by etching models in silicon

theregister.com

Post reply on HN
Print view View original

141–150 of 712 posts

Re: AMD acquires Taalas to boost inference performance by etching models in silicon

#141

Post by Melatonic »

mikeayles wrote: post #21

AMD could have saved their money and used their own hardware! I've got a language model doing 60k tok/s on AMD hardware already, a Xilinx Kria K26 SOM, with the weights baked into URAM/BRAM with zero DRAM in the token loop. Same thesis as Taalas: single-stream decode is bandwidth bound, so stop fetching weights from far away. Caveats stacked high, obviously. It's 3.16M parameters (tinystories, and I also have a kevin…

Yeah Im surprised nobody is talking about this. When everyone first saw Taalas I looked at the design and it had a big legup in physical cache availale compared to most chips. Makes you wonder how much of a benefit there is to the actual "baking" of the model vs just having a large chip with a ton of SRAM (or whatever) soldered close to the edge physically.

I feel like what we really need is the ability to solder computer cache on all sides of the chip Meaning above and below as well. If you can only attach it to the edges you will be inherently physically limited on the amount you can put (and maybe even have latency benefits as well)

Re: AMD acquires Taalas to boost inference performance by etching models in silicon

#142

Post by bastawhiz »

adgjlsfhk1 wrote: post #110
anthonypasq wrote: post #74

Earlier quoted context omitted.

Personally I think Apple should have acquired them. if you could burn a gemma4 class model into an iphone and actually get extremely low latency and low battery usage it would feel like the future IMO. even if it means you wont get frontier intelligence, there might actually be incentive to buy a new mobile device every year again.

I don't think this works out from a cost/silicon perspective. Small models already run pretty well in software (since the weights fit in cache) and big models require silicon area proportional to the size of weights. On a mobile device putting a chip like this is competing directly in BOM and power against a whole lot more l3 cache, and the l3 cache makes everything faster

The weights might fit in cache, if you're using a small model. If you wanted to have a 20B+ parameter model, that's just going in RAM. You could put more RAM in the device and pay the perf cost or have a dedicated chip. Most devices already have a dedicated chip, this just changes which silicon you're spending the money on.

Re: AMD acquires Taalas to boost inference performance by etching models in silicon

#143

Post by mdp2021 »

ggm wrote: post #137

Field reprogrammable, it's an FPGA on steroids. Field upgradable. Burnt in, it needs a zif socket and easy access in every car, aircraft, a pull out slot in a phone, or it's new era planned obselescence.

Can that be done when the whole idea is to store a multiplier into a handful of transistors?

Re: AMD acquires Taalas to boost inference performance by etching models in silicon

#144

Post by redox99 »

Is there any LLM from exactly one year ago that would be worth running?

In Aug 2025 you had

- OpenAI o3

- Opus 4.1

- Gemini 2.5 Pro

- Grok 4

Even if those were almost free to run, you'd be way better off with Deepseek flash 0731 or GPT 5.6 Luna, which already are almost free.

Other than for things where the t/s are critical, it seems like a bad idea to etch a model into silicon.

Re: AMD acquires Taalas to boost inference performance by etching models in silicon

#145

Post by IsTom »

cubefox wrote: post #123

> At 20 billion parameters per chip, you’d need just 50 accelerators to support a trillion-parameter model I don't see any evidence that this is possible. From my understanding, the whole model needs to be on a single chip. Which rules out any popular frontier models with several trillions of parameters. Even smaller sub-frontier models have hundreds of millions of parameters, so these would be ruled out as well.

I think it's enough that a single layer fits on each chip if you can daisy-chain them with good interconnects.

Re: AMD acquires Taalas to boost inference performance by etching models in silicon

#146

Post by wmf »

syntaxing wrote: post #20
VladVladikoff wrote: post #19

Earlier quoted context omitted.

Wouldn't this mean someone with sufficient hardware could lift the SOTA model weights off the chip? Or are you saying that these chips would only be used internally by these companies and not sold to the public?

I don’t get why this is an issue? You can run Claude/OpenAI SOTA models through Amazon bedrock. These weights have to live somewhere to run on Bedrock.

somewhere = an AWS data center with multiple layers of security and NDAs

They won't sell/rent/license the weights to an end user at any price because they don't trust your security.

Re: AMD acquires Taalas to boost inference performance by etching models in silicon

#147

Post by Gander5739 »

mdp2021 wrote: post #84
dumberquestions wrote: post #63

Earlier quoted context omitted.

So this demo is around 90 times faster than typical speeds for the same model at openrouter, and around 30 times faster than the absolute fastest option available (Groq).

https://taalas.com/h-content/uploads/2026/02/graph.png

https://xkcd.com/1162/

Re: AMD acquires Taalas to boost inference performance by etching models in silicon

#148

Post by RussianCow »

teaearlgraycold wrote: post #139
adgjlsfhk1 wrote: post #110

Earlier quoted context omitted.

I don't think this works out from a cost/silicon perspective. Small models already run pretty well in software (since the weights fit in cache) and big models require silicon area proportional to the size of weights. On a mobile device putting a chip like this is competing directly in BOM and power against a whole lot more l3 cache, and the l3 cache makes everything faster

My question is what changes about LLM use cases when you’re getting 1000 tok/s? Models in silicon might dramatically change how we think about them.

That likely isn't as relevant for on-device iPhone usage as it is for Real Work™. I won't notice the difference between 50tps and 1000tps when asking Siri a question.

Re: AMD acquires Taalas to boost inference performance by etching models in silicon

#149

Post by try-working »

moshun wrote: post #60
LarsDu88 wrote: post #44

I'm surprised neither OpenAI nor Anthropic made this move first. The Chinese open weight models are pulling ahead and commoditizing their value proposition. Baking models onto silicon would've been the next logical move to get a moat. Google is already doing this and has an experimental project on top of already having TPUs and cramming their quantized flash onto individual TPUs for inference.

Considering the rate of model development and rail hopping, seems like baking models into silicon is speed-running obsolescence.

obsolescence is the whole point. apple gets to sell a new phone very 6-12 months because of it.

i have written about this:

"For device makers

Packaging models with laptops and smartphones will let application access near free, low latency inference and potentially offer users a better experience with the option of preserving data on-device. This is viable under the condition that tasks that do require larger expert models that run in the cloud can be routed to external models. A side-effect of local models and what will let Apple cut upgrade cycles from ~4 years (?) down to 12-18 months is specialized hardware to run them. For almost a decade, smartphones have been trying to compete on better cameras. This coming decade will see them selling better GPUs, NPUs, ASICs and whatever other things they'll be calling the inference chips, to drive re-purchase. Every six months will see a better model on new hardware, which will enable better performance in certain applications."

https://try.works/role-model-the-case-for-a-model-routing-pr...

Re: AMD acquires Taalas to boost inference performance by etching models in silicon

#150

Post by zxspectrum1982 »

moshun wrote: post #60
LarsDu88 wrote: post #44

I'm surprised neither OpenAI nor Anthropic made this move first. The Chinese open weight models are pulling ahead and commoditizing their value proposition. Baking models onto silicon would've been the next logical move to get a moat. Google is already doing this and has an experimental project on top of already having TPUs and cramming their quantized flash onto individual TPUs for inference.

Considering the rate of model development and rail hopping, seems like baking models into silicon is speed-running obsolescence.

I'd gladly pay for a Claude Opus 4.6 Thinking High in silicon and use it for 1-2 years. It's good enough for many coding tasks.
Post reply on HN