Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

AMD acquires Taalas to boost inference performance by etching models in silicon

theregister.com

Post reply on HN
Print view View original

381–390 of 710 posts

Re: AMD acquires Taalas to boost inference performance by etching models in silicon

#381

Post by unsigner »

LarsDu88 wrote: post #44

I'm surprised neither OpenAI nor Anthropic made this move first. The Chinese open weight models are pulling ahead and commoditizing their value proposition. Baking models onto silicon would've been the next logical move to get a moat. Google is already doing this and has an experimental project on top of already having TPUs and cramming their quantized flash onto individual TPUs for inference.

Their thing is improving the models; it would be extremely counter-company-culture to bet on models plateau-ing. Maybe wise in terms of hedging, but still difficult to pull of as a company decision.

Re: AMD acquires Taalas to boost inference performance by etching models in silicon

#382

Post by AussieWog93 »

kllrnohj wrote: post #357
ilaksh wrote: post #278

Earlier quoted context omitted.

Cerebras already runs large models like Kimi 2.6 or GLM at like 30x speed. 100 times is next year, not six years. You can actually test it out on their website, just imagine 3 x faster and maybe 15% smarter.

Cerebras is literally the entire wafer, so it can't get bigger. So where is the jump from 30x to 100x coming from? Node improvements only yield like 10-20% gains these days...

Could we not just make bigger wafers, if the technology called for it?

Re: AMD acquires Taalas to boost inference performance by etching models in silicon

#383

Post by Iolaum »

subroutine wrote: post #228
zxspectrum1982 wrote: post #150

Earlier quoted context omitted.

I'd gladly pay for a Claude Opus 4.6 Thinking High in silicon and use it for 1-2 years. It's good enough for many coding tasks.

But Claude Opus 4.6 is not really practical. Taalas' process seems targeted for edge models. Their proof of concept model, for example, is a heavily quantized version of Llama 3.1 8B and even then they acknowledge their custom 3-bit/6-bit representation causes model quality degradation. Taalas is going to have a tough time putting a trillion-parameter model on one conventional die. Their HC1 die is already near the m…

A really fast qwen-3.6-27B type of model could be useful. With a specialized harness and this speed I 'd expect it to find many applications. Implementing a coding plan is the minimum I can think of.

Re: AMD acquires Taalas to boost inference performance by etching models in silicon

#384

Post by pstuart »

encyclopedism wrote: post #22
syntaxing wrote: post #11

Honestly, this is starting to make more and more sense. SOTA models are starting to converge to certain architecture and capabilities. I wouldn’t be surprised we end up with a base model ASIC + “fine tune” card where it’s a physical LoRA style adapter.

Imagine a multi-modal model with 1000's of tokens per second. Realtime inference for a host of applications. This is a BIG deal and will change the landscape in unfathomable ways. The https://chatjimmy.ai demo was impressive. Once models settle down this makes sense. Imagine a cartridge with a physical model on it. You purchase a cartridge and stick it in your computer/phone/server. Want to upgrade? By a new 'cartrid…

The cartridge could be a small mac-mini type unit connected and powered over thunderbolt. If it included like an m5 or m7 with 64GB of memory and a PCIe5/6 4TB Nvme it would be amazeballs. Hopefully when the bubble corrects and hardware advances and prices reset something like that will become available.

Just even comparing compute from 10 years ago (Apple silicon vs Intel) and it's significant. 20 years it gets crazy. My first computer was an 8 bit 6502 with 64K RAM and a 128K floppy drive (I think, it's fuzzy). Everything amazing now will look quaint in due time.

Re: AMD acquires Taalas to boost inference performance by etching models in silicon

#385

Post by manmal »

evilduck wrote: post #320
jodrellblank wrote: post #285

Earlier quoted context omitted.

> " The agent is running tests, loading webpages, etc, which all take time " A frustrating vision of the future would be when we've been asking for faster loading lighter web pages for years and then companies start caring about it and improving it not for us humans but for LLMs.

It's already kind of that way with MCP servers popping up everywhere. The JIRA MCP server is like a couple orders of magnitude faster to work with than the website itself.

That’s their API with extra steps, or am I missing something? That was always faster.

Re: AMD acquires Taalas to boost inference performance by etching models in silicon

#386

Post by itgoon »

redox99 wrote: post #144

Is there any LLM from exactly one year ago that would be worth running? In Aug 2025 you had - OpenAI o3 - Opus 4.1 - Gemini 2.5 Pro - Grok 4 Even if those were almost free to run, you'd be way better off with Deepseek flash 0731 or GPT 5.6 Luna, which already are almost free. Other than for things where the t/s are critical, it seems like a bad idea to etch a model into silicon.

Gemini 2.5 Pro was very good at writing single, somewhat complex functions. Sure, the rest of the loop would still take time, but nearly-instant implementation? Sign me up.

Re: AMD acquires Taalas to boost inference performance by etching models in silicon

#387

Post by onion2k »

chorizo wrote: post #257
Melatonic wrote: post #134

Earlier quoted context omitted.

The Taalas chips are not physically small. And part of their secret (if you look at the design) is just locating a bunch of memory soldered on the edges ( I belive higher amounts of SRAM ? )

Baking the base models on to ROM makes a lot of economic sense. SRAM for the KV cache & fine-tunes, not so much. Sure you’d get incredible speeds but it’s not scalable from a die-size or cost perspective. Rather base model on ROM + KV cache on DRAM is much more scalable. Also this would work great for edge devices that have a 2-5 year lifecycle.

Baking the base models on to ROM makes a lot of economic sense.

Less so for consumers though, because it'd mean the phone is out of date in 3 months when a better model comes along.

Re: AMD acquires Taalas to boost inference performance by etching models in silicon

#388

Post by manmal »

QuiDortDine wrote: post #251
teaearlgraycold wrote: post #139

Earlier quoted context omitted.

My question is what changes about LLM use cases when you’re getting 1000 tok/s? Models in silicon might dramatically change how we think about them.

Did you use chatjimmy? It's somewhat terrifying to use when you think of the potential results with a better model. Ok, real life example: I now spend most of my time, as a developer, waiting for the agent to do its thing (after careful prompting, I'm also thinking about work stuff, don't worry I'm not useless). What if it gave back the same excellent results, but instantaneously? Why, then, I certainly would become…

Errors compound, and making 1000 wrong decisions per hour, will not result in something useful. Maybe you‘ve tried setting up guardrails for good design or architecture at some point? I think it’s simply not possible to do that.

It would certainly be an accelerator for people who know exactly what they want. And it would remove multi tasking, which I‘d appreciate.

Re: AMD acquires Taalas to boost inference performance by etching models in silicon

#389

Post by ronsor »

onion2k wrote: post #387
chorizo wrote: post #257

Earlier quoted context omitted.

Baking the base models on to ROM makes a lot of economic sense. SRAM for the KV cache & fine-tunes, not so much. Sure you’d get incredible speeds but it’s not scalable from a die-size or cost perspective. Rather base model on ROM + KV cache on DRAM is much more scalable. Also this would work great for edge devices that have a 2-5 year lifecycle.

Baking the base models on to ROM makes a lot of economic sense. Less so for consumers though, because it'd mean the phone is out of date in 3 months when a better model comes along.

It's a perfect reason to get consumers to buy a new phone every year again! They got bored of the camera.

Re: AMD acquires Taalas to boost inference performance by etching models in silicon

#390

Post by kzrdude »

AussieWog93 wrote: post #382
kllrnohj wrote: post #357

Earlier quoted context omitted.

Cerebras is literally the entire wafer, so it can't get bigger. So where is the jump from 30x to 100x coming from? Node improvements only yield like 10-20% gains these days...

Could we not just make bigger wafers, if the technology called for it?

The investment in bigger machines at the fab might set you back billions. I don't know about the lithography technology either, how easy you can scale it to larger wafers?
Post reply on HN