Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

AMD acquires Taalas to boost inference performance by etching models in silicon

theregister.com

Post reply on HN
Print view View original

21–30 of 712 posts

Re: AMD acquires Taalas to boost inference performance by etching models in silicon

#21

Post by mikeayles »

AMD could have saved their money and used their own hardware! I've got a language model doing 60k tok/s on AMD hardware already, a Xilinx Kria K26 SOM, with the weights baked into URAM/BRAM with zero DRAM in the token loop. Same thesis as Taalas: single-stream decode is bandwidth bound, so stop fetching weights from far away.

Caveats stacked high, obviously. It's 3.16M parameters (tinystories, and I also have a kevin-speak lemmatised version), the tokens are characters, and the 60k record is 16 streams that each remember exactly one token of context, so it's blisteringly fast at saying nothing. The honest build with full context and KV caching still does ~19k tok/s on one stream though.

I keep messing with the blogpost with the live demo, but I'm planning on flipping it to live in the next day or two

Re: AMD acquires Taalas to boost inference performance by etching models in silicon

#22

Post by encyclopedism »

syntaxing wrote: post #11

Honestly, this is starting to make more and more sense. SOTA models are starting to converge to certain architecture and capabilities. I wouldn’t be surprised we end up with a base model ASIC + “fine tune” card where it’s a physical LoRA style adapter.

Imagine a multi-modal model with 1000's of tokens per second. Realtime inference for a host of applications. This is a BIG deal and will change the landscape in unfathomable ways.

The https://chatjimmy.ai demo was impressive.

Once models settle down this makes sense. Imagine a cartridge with a physical model on it. You purchase a cartridge and stick it in your computer/phone/server. Want to upgrade? By a new 'cartridge'.

This should bring inference cost down dramatically, I wonder how OpenAI/Anthropic feel about that.

Re: AMD acquires Taalas to boost inference performance by etching models in silicon

#24

Post by nurumaik »

_aavaa_ wrote: post #15
smokel wrote: post #13

Earlier quoted context omitted.

The technical aspects of SOTA models are not publicly documented. How do you know if something is converging?

If we had deepseek v4 flash 0731 etched on a chip it would be more than capable enough and fast enough for so many people's needs, even hardcore engineer.

Will be capable and fast enough for 2-3 weeks until new sota drops

Re: AMD acquires Taalas to boost inference performance by etching models in silicon

#26

Post by shaewest »

nsxwolf wrote: post #17
whythismatters wrote: post #6

The demo: https://chatjimmy.ai/

It doesn’t believe it’s running on that chip, it’s arguing with me

It's running a very small, non-reasoning model at the moment. But more generally, almost all LLMs argue on the hardware/model they are/are on.

Re: AMD acquires Taalas to boost inference performance by etching models in silicon

#27

Post by cmrdporcupine »

bhouston wrote: post #12

Toronto Canada startup btw.

Seems to be somehow some kind of offshoot from or connected to Tenstorrent, which is just down the road. Founder looks like he was/is maybe at Tenstorrent and previously associated with Keller?

Always fantasize about applying at Tenstorrent, but wrong side of Toronto. 2 hour commute.

Re: AMD acquires Taalas to boost inference performance by etching models in silicon

#28

Post by snek_case »

VladVladikoff wrote: post #19
syntaxing wrote: post #11

Honestly, this is starting to make more and more sense. SOTA models are starting to converge to certain architecture and capabilities. I wouldn’t be surprised we end up with a base model ASIC + “fine tune” card where it’s a physical LoRA style adapter.

Wouldn't this mean someone with sufficient hardware could lift the SOTA model weights off the chip? Or are you saying that these chips would only be used internally by these companies and not sold to the public?

The weights are very unlikely to be on the chip itself. That wouldn't work for SOTA models that are terabyte scale, even quantized. This is probably an accelerator for specific kernels in the model, but the weights are likely loaded from memory. The chip may have SRAM to store some of the weights temporarily during inference.

Re: AMD acquires Taalas to boost inference performance by etching models in silicon

#29

Post by amazingamazing »

VladVladikoff wrote: post #19
syntaxing wrote: post #11

Honestly, this is starting to make more and more sense. SOTA models are starting to converge to certain architecture and capabilities. I wouldn’t be surprised we end up with a base model ASIC + “fine tune” card where it’s a physical LoRA style adapter.

Wouldn't this mean someone with sufficient hardware could lift the SOTA model weights off the chip? Or are you saying that these chips would only be used internally by these companies and not sold to the public?

One idea would be to use an open model.
Post reply on HN