Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

AMD acquires Taalas to boost inference performance by etching models in silicon

theregister.com

Post reply on HN
Print view View original

331–340 of 712 posts

Re: AMD acquires Taalas to boost inference performance by etching models in silicon

#331

Post by andersonpico »

jodrellblank wrote: post #285
xur17 wrote: post #274

Earlier quoted context omitted.

I'm not sure inference speed is always the slowest thing for me right now. The agent is running tests, loading webpages, etc, which all take time. I don't know if a fast agent would speed things up in all cases. That said, it obviously depends on the project.

> " The agent is running tests, loading webpages, etc, which all take time " A frustrating vision of the future would be when we've been asking for faster loading lighter web pages for years and then companies start caring about it and improving it not for us humans but for LLMs.

They finally cared about clear requirements and documentation when that meant getting rid of devs.

Re: AMD acquires Taalas to boost inference performance by etching models in silicon

#332

Post by adrianN »

chorizo wrote: post #257
Melatonic wrote: post #134

Earlier quoted context omitted.

The Taalas chips are not physically small. And part of their secret (if you look at the design) is just locating a bunch of memory soldered on the edges ( I belive higher amounts of SRAM ? )

Baking the base models on to ROM makes a lot of economic sense. SRAM for the KV cache & fine-tunes, not so much. Sure you’d get incredible speeds but it’s not scalable from a die-size or cost perspective. Rather base model on ROM + KV cache on DRAM is much more scalable. Also this would work great for edge devices that have a 2-5 year lifecycle.

It is my understanding that just baking the model itself into silicon only gives moderate gains because memory bandwidth remains a bottleneck.

Re: AMD acquires Taalas to boost inference performance by etching models in silicon

#333

Post by fwip »

anigbrowl wrote: post #306
teraflop wrote: post #222

Earlier quoted context omitted.

True, but predictability cuts both ways. We're all used to having to constantly update our browsers and phones to keep up with the security arms race. If a frozen model can't be updated, it will predictably remain vulnerable to any "exploits" or idiosyncratic quirks that people discover over time. Let's say, as somebody suggested in another comment, that you buy 100,000 of these chips and deploy them to run fast-food…

I see your argument but your example seems highly contrived. I can't think why you'd want to use something like this for something as dynamic as takeout ordering, where you might have to deal with bad customers, supply chain breakages, public health recalls, or any of many other probabilistic events. I think it's far more likely to see them used in safety critical applications where you need a capable model that can…

What safety critical applications would be a good fit for LLMs?

Re: AMD acquires Taalas to boost inference performance by etching models in silicon

#334

Post by msdz »

XCSme wrote: post #180
VBprogrammer wrote: post #56

Earlier quoted context omitted.

I had the same reaction but then I showed it to my partner. She completely didn't get it, in her words "how can it be thinking of a good answer when it's that quick?" I tried to explain but I fear were probably going to be adding artificial sleeps to these things to convince the masses it's doing something clever.

Wait, is it even thinking? Or is it an instant model?

It’s not reasoning, the hardware demo uses a 3.-something generation Llama 8B.

But it’s proven they can automate this (they didn’t etch eight billion weights by hand after all, obviously), so now the interesting question is whether they can scale it to more recent aka bigger models.

After all, there’s already very useful models even for productivity at 27 or 35B.

Re: AMD acquires Taalas to boost inference performance by etching models in silicon

#335

Post by [deleted] »

whatsThisBtn4 wrote: post #135
anthonypasq wrote: post #74

Earlier quoted context omitted.

Personally I think Apple should have acquired them. if you could burn a gemma4 class model into an iphone and actually get extremely low latency and low battery usage it would feel like the future IMO. even if it means you wont get frontier intelligence, there might actually be incentive to buy a new mobile device every year again.

[flagged]

[deleted]

Re: AMD acquires Taalas to boost inference performance by etching models in silicon

#336

Post by chorizo »

adrianN wrote: post #332
chorizo wrote: post #257

Earlier quoted context omitted.

Baking the base models on to ROM makes a lot of economic sense. SRAM for the KV cache & fine-tunes, not so much. Sure you’d get incredible speeds but it’s not scalable from a die-size or cost perspective. Rather base model on ROM + KV cache on DRAM is much more scalable. Also this would work great for edge devices that have a 2-5 year lifecycle.

It is my understanding that just baking the model itself into silicon only gives moderate gains because memory bandwidth remains a bottleneck.

The big benefit is ROM cells require fewer components than DRAM. So the chips would be tiny, dense, cheap and consume far less power.

Re: AMD acquires Taalas to boost inference performance by etching models in silicon

#337

Post by sanex »

bigyabai wrote: post #323
barbazoo wrote: post #293

Earlier quoted context omitted.

With more wealth concentrates at the top, yes.

Presumably wealth would concentrate upwards even if AI was never made.

Yes it's a function of the monetary system. Absurd amounts of debt only certain people can access.

Re: AMD acquires Taalas to boost inference performance by etching models in silicon

#338

Post by zkmon »

I guess the idea is, gains from inference speed could offset the cost of upgrading the chips to a new model when really required. I think general purpose models would consolidate and release frequency might flatten out, favoring this strategy.

Re: AMD acquires Taalas to boost inference performance by etching models in silicon

#340

Post by jodrellblank »

senderista wrote: post #50
whythismatters wrote: post #6

The demo: https://chatjimmy.ai/

Wow, feels like Google web search in 1999.

or LiveGrep fast search of the Linux kernel source code with regex support: https://livegrep.com/search/linux
Post reply on HN