Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

AMD acquires Taalas to boost inference performance by etching models in silicon

theregister.com

Post reply on HN
Print view View original

221–230 of 712 posts

Re: AMD acquires Taalas to boost inference performance by etching models in silicon

#221

Post by christina97 »

yumraj wrote: post #121

Given the fast churn of the models, how does it work out? Won’t the silicon etched model already be 1 or more versions behind by the time the silicon comes out. Though if it’s cheap enough, there certainly can be a market for cheaper model inferences.

There’s some kind of tradeoff between speed, cost, and quality for every application. I would be perfectly happy with a model 6 months old that was 50x faster for many uses. Right now I use either Opus (for smart stuff) or Flash without thinking (for fast stuff). I would take an even dumber model for more speed (lower latency in particular).

Re: AMD acquires Taalas to boost inference performance by etching models in silicon

#222

Post by teraflop »

anigbrowl wrote: post #129
wolttam wrote: post #77

Earlier quoted context omitted.

It's a terrible moat. You etch the silicon then nobody wants to run it in 6 months because models have advanced that much further.

This is only true for people who are solely focused on performance. There is absolutely a market for acceptable performance combined with predictability.

True, but predictability cuts both ways.

We're all used to having to constantly update our browsers and phones to keep up with the security arms race. If a frozen model can't be updated, it will predictably remain vulnerable to any "exploits" or idiosyncratic quirks that people discover over time.

Let's say, as somebody suggested in another comment, that you buy 100,000 of these chips and deploy them to run fast-food drive-thrus. And then somebody discovers the model has a fondness for goblins[1], and if you role-play convincingly enough, you can get it to accept payment in shiny buttons and rodent skulls instead of cash.

What do you do then? I guess your options are to try and fix the behavior with a better prompt, or put some kind of filter in front of the model to catch attempted exploits. If the filter is cheap and dumb it probably won't work well enough, and if you use another model as a filter, you've negated the cost and speed benefits of putting the first model in hardware.

Of course the real answer is to just never expose the model to situations where an adversarial input could possibly lead to an undesired output. But that drastically limits what you can do with it.

[1]: https://openai.com/index/where-the-goblins-came-from/

Re: AMD acquires Taalas to boost inference performance by etching models in silicon

#223

Post by jaggederest »

Melatonic wrote: post #190
Legend2440 wrote: post #163

Earlier quoted context omitted.

What you're describing is what Cerberas does. Talaas is different, it's a true compute-in-memory architecture where the weights are stored in the connections between the transistors that perform the matrix multiply, rather than in seperate memory cells. Most of the benefit comes from this architecture; hardwiring the weights into the silicon is just the easiest way to implement it. SRAM requires too many transistors,…

Cerebras chips are massive and do have more on the edge but they dont have any top or bottom cache do they?

They can't due to power density, I believe - they have to be run in a sandwiched waterblock with massive cooling, as far as I can tell. That's the biggest thing that baked weights gets you - a relatively modest watts-per-square-mm compare to cerebras, where they had to engineer a whole system to get the watts out of the chip

Re: AMD acquires Taalas to boost inference performance by etching models in silicon

#224

Post by ricardobeat »

varun_ch wrote: post #98
VBprogrammer wrote: post #56

Earlier quoted context omitted.

I had the same reaction but then I showed it to my partner. She completely didn't get it, in her words "how can it be thinking of a good answer when it's that quick?" I tried to explain but I fear were probably going to be adding artificial sleeps to these things to convince the masses it's doing something clever.

to be fair, the model used for Chat Jimmy is not very smart, but the world where it is smart is very interesting. It’s going to be really crazy when the bottle neck for agents is the speed of the tool calls rather than the speed of inference. Imagine an agent interacting with the terminal near instantly…

I had the chance to try out MiMo v2.5 Pro Ultraspeed (600-1000tok/s) for a couple weeks and it is amazing.

Developing software becomes 95% about intent and requirements. Can’t wait for the next iteration of that.

Re: AMD acquires Taalas to boost inference performance by etching models in silicon

#228

Post by subroutine »

zxspectrum1982 wrote: post #150
moshun wrote: post #60

Earlier quoted context omitted.

Considering the rate of model development and rail hopping, seems like baking models into silicon is speed-running obsolescence.

I'd gladly pay for a Claude Opus 4.6 Thinking High in silicon and use it for 1-2 years. It's good enough for many coding tasks.

But Claude Opus 4.6 is not really practical. Taalas' process seems targeted for edge models. Their proof of concept model, for example, is a heavily quantized version of Llama 3.1 8B and even then they acknowledge their custom 3-bit/6-bit representation causes model quality degradation.

Taalas is going to have a tough time putting a trillion-parameter model on one conventional die. Their HC1 die is already near the maximum size that conventional lithography can expose. They claim they could partition the model across many chips, but I'm not sure if they have tested this process or what it means for compute. The basic storage arithmetic is unforgiving: for a one trillion parameters model at four bits it will take 50–100 chips. To service a sizable customer base will take thousands of 100-chip fabs.

That all said, I'm bullish on this technology, and look forward to seeing it evolve.

Re: AMD acquires Taalas to boost inference performance by etching models in silicon

#229

Post by m463 »

XCSme wrote: post #185
yumraj wrote: post #121

Given the fast churn of the models, how does it work out? Won’t the silicon etched model already be 1 or more versions behind by the time the silicon comes out. Though if it’s cheap enough, there certainly can be a market for cheaper model inferences.

I think this would make sense for consumer hardware, not for AI companies. AI companies constantly update/change stuff, new models come out, new requirements, etc. But if you ship an "ai-powered" dishwasher, it can come with the chip built-in to do computer vision and precisely target each spot, and will be sold as-is with no updates.

subscription "ai-powered" dishwasher with personalized user ads, most of the chip dedicated to "personalized" not spots.

Re: AMD acquires Taalas to boost inference performance by etching models in silicon

#230

Post by craftkiller »

yumraj wrote: post #121

Given the fast churn of the models, how does it work out? Won’t the silicon etched model already be 1 or more versions behind by the time the silicon comes out. Though if it’s cheap enough, there certainly can be a market for cheaper model inferences.

I think the real value here is not as a customer-facing agent/chatbot but for for automated processes. Think of all the companies out there that have LLMs doing simple tasks like categorizing customer feedback emails. For such tasks, you don't gain much from better models, so if you could run it 10x cheaper on a slightly older model, it would absolutely be worth it. Pretty much any place people are currently running a flash model could benefit from this since they're already deciding that speed+price is worth using a less capable model.
Post reply on HN