Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

AMD acquires Taalas to boost inference performance by etching models in silicon

theregister.com

Post reply on HN
Print view View original

51–60 of 712 posts

Re: AMD acquires Taalas to boost inference performance by etching models in silicon

#51

Post by Grosvenor »

encyclopedism wrote: post #22
syntaxing wrote: post #11

Honestly, this is starting to make more and more sense. SOTA models are starting to converge to certain architecture and capabilities. I wouldn’t be surprised we end up with a base model ASIC + “fine tune” card where it’s a physical LoRA style adapter.

Imagine a multi-modal model with 1000's of tokens per second. Realtime inference for a host of applications. This is a BIG deal and will change the landscape in unfathomable ways. The https://chatjimmy.ai demo was impressive. Once models settle down this makes sense. Imagine a cartridge with a physical model on it. You purchase a cartridge and stick it in your computer/phone/server. Want to upgrade? By a new 'cartrid…

> Imagine a cartridge with a physical model on it.

I can finally have my own Dixie flatline. Cool.

Re: AMD acquires Taalas to boost inference performance by etching models in silicon

#52

Post by nojs »

Can anyone comment on the economics and likely turnaround times of this process, when it’s more mature?

Would it be realistic for a frontier lab to deploy this or would the turnaround time mean the model is always too out of date?

Assuming the weights and architecture are eventually stable, how much cheaper would this end up being?

Re: AMD acquires Taalas to boost inference performance by etching models in silicon

#53

Post by questionableans »

tandr wrote: post #38
mikeayles wrote: post #21

AMD could have saved their money and used their own hardware! I've got a language model doing 60k tok/s on AMD hardware already, a Xilinx Kria K26 SOM, with the weights baked into URAM/BRAM with zero DRAM in the token loop. Same thesis as Taalas: single-stream decode is bandwidth bound, so stop fetching weights from far away. Caveats stacked high, obviously. It's 3.16M parameters (tinystories, and I also have a kevin…

Well, technically it is their hardware now...

And their team, if they treat them well.

Re: AMD acquires Taalas to boost inference performance by etching models in silicon

#54

Post by walrus01 »

syntaxing wrote: post #11

Honestly, this is starting to make more and more sense. SOTA models are starting to converge to certain architecture and capabilities. I wouldn’t be surprised we end up with a base model ASIC + “fine tune” card where it’s a physical LoRA style adapter.

Having a base model ASIC as a physical piece of hardware makes me think of the early days of microcomputer desktop stuff where having a socketed ROM or PROM was a key piece of hardware, and people actually knew/cared what ROM was on their system's motherboard.

Imagine if like instead of having a specific Mac Plus ROM, you had a thing that looks like a fat ASIC that can hold models sitting on a slotted daughtercard directly next to the CPU and RAM.

Re: AMD acquires Taalas to boost inference performance by etching models in silicon

#55

Post by thoughtpeddler »

walrus01 wrote: post #47
whythismatters wrote: post #6

The demo: https://chatjimmy.ai/

I know it's a relatively tiny model, but damn, is that thing fast. It also mostly passes the "schlong" test https://pastes.io/YcxSi8Fp

I didn't realize there was a SchlongBench™ (but of course there is). What's it test? (asking seriously)

Re: AMD acquires Taalas to boost inference performance by etching models in silicon

#56

Post by VBprogrammer »

wxw wrote: post #31
whythismatters wrote: post #6

The demo: https://chatjimmy.ai/

I freakin' love this demo. It feels magical.

I had the same reaction but then I showed it to my partner. She completely didn't get it, in her words "how can it be thinking of a good answer when it's that quick?"

I tried to explain but I fear were probably going to be adding artificial sleeps to these things to convince the masses it's doing something clever.

Re: AMD acquires Taalas to boost inference performance by etching models in silicon

#57

Post by penagwin »

metadat wrote: post #46
shaewest wrote: post #26

Earlier quoted context omitted.

It's running a very small, non-reasoning model at the moment. But more generally, almost all LLMs argue on the hardware/model they are/are on.

What would tokens/sec performance look like for a reasoning model? An order of magnitude slower?

Reasoning models are the same speed. They’re just post trained with RL to do CoT inside tags like before a tag like

There’s no difference in the inference implementation, parameter count, or speed.

Re: AMD acquires Taalas to boost inference performance by etching models in silicon

#59

Post by LPisGood »

LarsDu88 wrote: post #44

I'm surprised neither OpenAI nor Anthropic made this move first. The Chinese open weight models are pulling ahead and commoditizing their value proposition. Baking models onto silicon would've been the next logical move to get a moat. Google is already doing this and has an experimental project on top of already having TPUs and cramming their quantized flash onto individual TPUs for inference.

I’m surprised Nvidia hasn’t partnered to make a Claude chip yet. It’s a win/win you can license them out, sell them when they become obsolete, etc.

Re: AMD acquires Taalas to boost inference performance by etching models in silicon

#60

Post by moshun »

LarsDu88 wrote: post #44

I'm surprised neither OpenAI nor Anthropic made this move first. The Chinese open weight models are pulling ahead and commoditizing their value proposition. Baking models onto silicon would've been the next logical move to get a moat. Google is already doing this and has an experimental project on top of already having TPUs and cramming their quantized flash onto individual TPUs for inference.

Considering the rate of model development and rail hopping, seems like baking models into silicon is speed-running obsolescence.
Post reply on HN