Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

AMD acquires Taalas to boost inference performance by etching models in silicon

theregister.com

Post reply on HN
Print view View original

231–240 of 712 posts

Re: AMD acquires Taalas to boost inference performance by etching models in silicon

#231

Post by SR2Z »

giancarlostoro wrote: post #217
SR2Z wrote: post #183

Earlier quoted context omitted.

> cheaper in all ways, Bitcoin mining doesn't have large memory requirements, but does have huge compute requirements. ASICs work great there because it's very straightforward to add some circuits for computing hashes. If you _also_ have to add many GB of memory, then suddenly ASICs will cost as much or more than comparable off-the-shelf hardware and they won't be faster unless you've also invested in huge memory ban…

My understanding is an ASIC can last 10+ years, where are Nvidia enterprise GPUs are rated for 5...

Most enterprise GPUs are scrap after 5 years because they're so inefficient compared to newer models. It's entirely possible to make them last longer by undervolting them, people just don't because it doesn't make sense.

Bitcoin OTOH has used the same PoW algorithm for a decade. Barring some really exciting discoveries about the nature of computation, new ASICs are not that much more efficient than old ones.

BTC mining is also not exactly competitive anymore; the nature of the PoW algorithm means that it's dominated by a few large players who've set up shop next to a dam and who pay very little for electricity.

New entrants are highly discouraged because the mining rewards are constantly halving, it's hard to find cheap power, and the price of BTC is now so volatile that a yearslong investment is very likely to lose money.

Re: AMD acquires Taalas to boost inference performance by etching models in silicon

#232

Post by mdp2021 »

foltik wrote: post #115
snek_case wrote: post #28

Earlier quoted context omitted.

The weights are very unlikely to be on the chip itself. That wouldn't work for SOTA models that are terabyte scale, even quantized. This is probably an accelerator for specific kernels in the model, but the weights are likely loaded from memory. The chip may have SRAM to store some of the weights temporarily during inference.

At least in the case of Taalas the weights are physically encoded directly on the chip. It’s composed of 4-bit multiplier cells that compute all 16 possible results in parallel. The top metal wiring layer physically selects the one that corresponds to a multiplication with that cell’s constant weight, and routes it to the next layer.

Are you sure? Source? (does not seem to be https://taalas.com/the-path-to-ubiquitous-ai/ , for example)

Re: AMD acquires Taalas to boost inference performance by etching models in silicon

#233

Post by dabbz »

retatop wrote: post #194
RussianCow wrote: post #148

Earlier quoted context omitted.

That likely isn't as relevant for on-device iPhone usage as it is for Real Work™. I won't notice the difference between 50tps and 1000tps when asking Siri a question.

But wouldn't higher tps allow for more reasoning or other hidden processes, potententially making a smarter model?

This is my thought as well. Models have to be intentional about which tokens they burn because there's a real lag time. If you can just fork out 10 different reasoning sessions at once with no regard for token waste/lag, you can compensate a smaller model with just doing more at once with it. No idea if this is reasonably true though.

Re: AMD acquires Taalas to boost inference performance by etching models in silicon

#235

Post by sgc »

What does it take to go from here to a model on a pcie card or an m.2 card, so I can plug one into my workstation / laptop? Will 'intelligence' become much like a gpu, where most people just live with the performance of whatever they have installed, outside large companies that must have cutting edge, or prosumers that have a incrementally better version than the masses?

Are we a couple years away, a decade away, or something else?

Re: AMD acquires Taalas to boost inference performance by etching models in silicon

#236

Post by mdp2021 »

Gigachad wrote: post #173
zxspectrum1982 wrote: post #150

Earlier quoted context omitted.

I'd gladly pay for a Claude Opus 4.6 Thinking High in silicon and use it for 1-2 years. It's good enough for many coding tasks.

It costs something like $300,000 for the hardware to run a model of that size. You'd pay that for a single model for 1-2 years? Not even the AI companies can justify that kind of spend which is why they keep extending the expected lifespan on their hardware in the accounting.

> It costs something like $300,000 for the hardware to run a model of that size

You did not compute that as the cost for a speculative card from Taalas, right?

Re: AMD acquires Taalas to boost inference performance by etching models in silicon

#237

Post by HDBaseT »

teaearlgraycold wrote: post #139
adgjlsfhk1 wrote: post #110

Earlier quoted context omitted.

I don't think this works out from a cost/silicon perspective. Small models already run pretty well in software (since the weights fit in cache) and big models require silicon area proportional to the size of weights. On a mobile device putting a chip like this is competing directly in BOM and power against a whole lot more l3 cache, and the l3 cache makes everything faster

My question is what changes about LLM use cases when you’re getting 1000 tok/s? Models in silicon might dramatically change how we think about them.

In the case on on-device/self-hosted LLMs. You ask your agent to implement xyz feature 10 times and use a model to compare the outputs and combine the best results.

Raw intelligence becomes slightly less important when you can iterate and improve automatically. You can still claim it was "one shot" even when 30 different implementations were made then combined.

Re: AMD acquires Taalas to boost inference performance by etching models in silicon

#239

Post by mdp2021 »

sgc wrote: post #235

What does it take to go from here to a model on a pcie card or an m.2 card, so I can plug one into my workstation / laptop? Will 'intelligence' become much like a gpu, where most people just live with the performance of whatever they have installed, outside large companies that must have cutting edge, or prosumers that have a incrementally better version than the masses? Are we a couple years away, a decade away, or…

> What does it take to go from here to a model on a pcie card or an m.2 card

It is already that.

> Will "intelligence" become much like a gpu

As an option among the implementations.

> Are we a couple years away

They could mass produce now, but it makes no sense at this rate of improvements in the models.

Re: AMD acquires Taalas to boost inference performance by etching models in silicon

#240

Post by wmf »

bastawhiz wrote: post #142
adgjlsfhk1 wrote: post #110

Earlier quoted context omitted.

I don't think this works out from a cost/silicon perspective. Small models already run pretty well in software (since the weights fit in cache) and big models require silicon area proportional to the size of weights. On a mobile device putting a chip like this is competing directly in BOM and power against a whole lot more l3 cache, and the l3 cache makes everything faster

The weights might fit in cache, if you're using a small model. If you wanted to have a 20B+ parameter model, that's just going in RAM. You could put more RAM in the device and pay the perf cost or have a dedicated chip. Most devices already have a dedicated chip, this just changes which silicon you're spending the money on.

That math doesn't really work.

8B model (FP4) = 4 GB DRAM = 32 Gb DRAM = 80 mm2

8B model (Taalas) = 4 GB ROM = ~800 mm2

Post reply on HN