Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

AMD acquires Taalas to boost inference performance by etching models in silicon

theregister.com

Post reply on HN
Print view View original

241–250 of 712 posts

Re: AMD acquires Taalas to boost inference performance by etching models in silicon

#241

Post by andix »

walrus01 wrote: post #47
whythismatters wrote: post #6

The demo: https://chatjimmy.ai/

I know it's a relatively tiny model, but damn, is that thing fast. It also mostly passes the "schlong" test https://pastes.io/YcxSi8Fp

It failed on my usual test. But it failed really fast:

"A farmer has a wolf, a goat, and a cabbage. The wolf is imaginary and doesn't exist. He wants to cross the river, but the boat is only big enough to hold him and one of them. The farmer can't leave the wolf and the goat together, because the wolf will eat the goat. Similarly, he can't leave the goat and the cabbage together, because the goat will eat the cabbage. What is the smallest number of trips the farmer needs to make to get everything across the river?"

Re: AMD acquires Taalas to boost inference performance by etching models in silicon

#242

Post by heywoods »

topspin wrote: post #119
moshun wrote: post #60

Earlier quoted context omitted.

Considering the rate of model development and rail hopping, seems like baking models into silicon is speed-running obsolescence.

"seems like baking models into silicon is speed-running obsolescence" Now maybe. When models are flying passenger aircraft, other prerogatives will assert themselves. When a 50TB ROM means you can impulse purchase a ChatGPT 6.3 xhigh that runs on batteries, yet more use cases will be apparent.

Or autonomous weapon systems, missiles, and drones.

Re: AMD acquires Taalas to boost inference performance by etching models in silicon

#243

Post by appplication »

whythismatters wrote: post #6

The demo: https://chatjimmy.ai/

This is the coolest LLM thing I’ve seen since the original ChatGPT announcement a few years ago. IMO much more impressive than marginal gains of frontier models.

Re: AMD acquires Taalas to boost inference performance by etching models in silicon

#244

Post by dboreham »

adgjlsfhk1 wrote: post #110
anthonypasq wrote: post #74

Earlier quoted context omitted.

Personally I think Apple should have acquired them. if you could burn a gemma4 class model into an iphone and actually get extremely low latency and low battery usage it would feel like the future IMO. even if it means you wont get frontier intelligence, there might actually be incentive to buy a new mobile device every year again.

I don't think this works out from a cost/silicon perspective. Small models already run pretty well in software (since the weights fit in cache) and big models require silicon area proportional to the size of weights. On a mobile device putting a chip like this is competing directly in BOM and power against a whole lot more l3 cache, and the l3 cache makes everything faster

Works great from a press release perspective though.

Re: AMD acquires Taalas to boost inference performance by etching models in silicon

#245

Post by wmf »

XCSme wrote: post #192
ggm wrote: post #137

Field reprogrammable, it's an FPGA on steroids. Field upgradable. Burnt in, it needs a zif socket and easy access in every car, aircraft, a pull out slot in a phone, or it's new era planned obselescence.

Why not have some a device/hardware that programs itself on-boot. Sort of a FPGA, that (electrically) arranges the connections on-boot, and then it's like a static inference chip.

FPGAs already configure themselves on boot.

Re: AMD acquires Taalas to boost inference performance by etching models in silicon

#246

Post by andix »

zxspectrum1982 wrote: post #150
moshun wrote: post #60

Earlier quoted context omitted.

Considering the rate of model development and rail hopping, seems like baking models into silicon is speed-running obsolescence.

I'd gladly pay for a Claude Opus 4.6 Thinking High in silicon and use it for 1-2 years. It's good enough for many coding tasks.

With thousands of token per second output it would be an enormous waste of resources. Such chips are clearly made to process thousands of conversations simultaneously. Not necessarily in parallel. All LLM workflows are turn based right now, there are often seconds between turns until tool calls finish or users type the next message.

If the LLM response only takes a few milliseconds, the chip can process hundreds of other requests until the first conversation becomes active again.

Re: AMD acquires Taalas to boost inference performance by etching models in silicon

#247

Post by krisoft »

makeitdouble wrote: post #209
anthonypasq wrote: post #74

Earlier quoted context omitted.

Personally I think Apple should have acquired them. if you could burn a gemma4 class model into an iphone and actually get extremely low latency and low battery usage it would feel like the future IMO. even if it means you wont get frontier intelligence, there might actually be incentive to buy a new mobile device every year again.

Slightly besides your point, but it's interesting how many here naturally ponder about how the current winner could or "should" keep winning, instead of how another company could become a competitor by doing the more clever thing the incumbent isn't thinking about.

It is not a “should”. At least not in the “we wish it were so” sense.

It is more that there are multiple reasons why this idea (burning an LLM into silicone and deploying it into a device in people’s pockets) requires huge piles of cash and the kind of engineering chops only a few company posesses.

Of course i would like it if a small upstart would do this, but it doesn’t seem likely as a posibility. They won’t have the funds to fab the IC. They won’t have the funds to train and validate the model before burning it into silicone. They can’t absorb the risk of the first tape out going wrong. They can’t absorb the risk of the model being faulty in some subtle way. They don’t have a device to integrate the IC into. They won’t have the funds to develop one. If they somehow would make a device they don’t have the marketing and sales channels built out to get the device into people’s hands in sufficient numbers to justify the development cost.

Basically this idea feels ruinously expensive. Apple has deep pockets, they already have working well-regarded phones, and an ethos of privacy preserving innovation. This is why this idea feels well suited for them and not many others.

Do i want the winners to keep winning? No. But not many others can pay for a moonshot crossed with a manhattan project. They just can’t.

Re: AMD acquires Taalas to boost inference performance by etching models in silicon

#248

Post by AlotOfReading »

teaearlgraycold wrote: post #140
mdp2021 wrote: post #125

Earlier quoted context omitted.

> A small model would be [mobile size] A ~30mm side for the HC1 tech for an 8b model (still unclear the planned HC2)?

Is that analogue or are they baking floating points into the silicon?

It's entirely possible they're using something like block floating point, where most of the hardware is simply fixed point. AMD's NPU does this, for example.

Re: AMD acquires Taalas to boost inference performance by etching models in silicon

#249

Post by roughly »

drchickensalad wrote: post #218
roughly wrote: post #204

How's that jive with the fact that they're introducing a new model every other week?

The new model every week is not necessary at this point really. What if you could run opus 5 for the next couple years at 1/20 the cost?

What's interesting about this is that I as a user would find this useful, but I think the AI industry as a whole would find it an absolute goddamn disaster. Opus 5 is a very good tool, but it is not a human-replacement-level intelligence, which means the entire revenue stream the industry's built on - labor replacement - is not met by this, and the only slightly charitable read of the industry's finances is that they're gonna bootstrap their way to creating the labor replacement hypothesis by getting people to spend money on Opus/etc, whereas if the actual product is a 1/20th the cost Opus-on-a-chip, the entire business and financing model that's tying up $N Trillion dollars of investment money goes out the window.

Great for us, looks like a recession as far as the Market is concerned.

Re: AMD acquires Taalas to boost inference performance by etching models in silicon

#250

Post by intrasight »

spijdar wrote: post #179
RussianCow wrote: post #148

Earlier quoted context omitted.

That likely isn't as relevant for on-device iPhone usage as it is for Real Work™. I won't notice the difference between 50tps and 1000tps when asking Siri a question.

I don't know. As others have said, the Taalas chip wasn't small, or particularly low power, so it's hard to "imagine" what that tech in an cell phone chip might look like. But if the basic premise of "good enough LLM at insane throughput" holds, I think it could qualitatively change local uses of LLMs. At a certain speed point, you're able to move from request -> response to a cascade of tool calling and "subagents",…

Box that plugs into my desktop would be fine. Or perhaps in SSF form factor.
Post reply on HN