Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

AMD acquires Taalas to boost inference performance by etching models in silicon

theregister.com

Post reply on HN
Print view View original

661–670 of 712 posts

Re: AMD acquires Taalas to boost inference performance by etching models in silicon

#661

Post by try-working »

nomel wrote: post #637
try-working wrote: post #360

Earlier quoted context omitted.

you don't understand what I wrote.

I do. The point is inference speed and power, making previously impossible local inference possible. A side effect of that hardware optimization is fixed capabilities. You've confused engineering compromise for malice, and reversed the purpose. For the model capabilities and inference power draw, what alternative do you see to a (at least mostly) fixed hardware model?

What I'm saying is that Apple will use these type of models etched into chips, and they will do it because it drives obsolescence, so they can shorten the upgrade cycle. They will do it because they figure out it's good for them.

Re: AMD acquires Taalas to boost inference performance by etching models in silicon

#662

Post by fph »

v9v wrote: post #494
logicallee wrote: post #478

Earlier quoted context omitted.

if it's baked into silicon how can you two get different answers?

It still works the same way other LLMs do, by outputting the probability distribution over the possible completions (The weather is ... (sunny (50%), cloudy (50%))). Then the next token is sampled from this probability distribution (in our example the next word could be "sunny" or "cloudy" equally likely), which can result in different outputs every run.

But since it's so fast you can just ask it 100 times where Bruce Lee was born, and statistically you'll get the correct answer. We could call it "mixture of idiots". /s

Re: AMD acquires Taalas to boost inference performance by etching models in silicon

#663

Post by [deleted] »

try-working wrote: post #661
nomel wrote: post #637

Earlier quoted context omitted.

I do. The point is inference speed and power, making previously impossible local inference possible. A side effect of that hardware optimization is fixed capabilities. You've confused engineering compromise for malice, and reversed the purpose. For the model capabilities and inference power draw, what alternative do you see to a (at least mostly) fixed hardware model?

What I'm saying is that Apple will use these type of models etched into chips, and they will do it because it drives obsolescence, so they can shorten the upgrade cycle. They will do it because they figure out it's good for them.

[deleted]

Re: AMD acquires Taalas to boost inference performance by etching models in silicon

#664

Post by cptskippy »

orangeberrytea wrote: post #603
TechTechTech wrote: post #518

I think what will happen is what happened to something like 4K video decoding before where it ends up in silicon costing almost nothing to run extremely fast on device. "Good enough" LLM functionality (for the use case) will be on-die or on-chip for cars, appliances, etc. This will provide speeds of chatjimmy at a battery-level power consumption. Probably this will also happen for software engineering. Some usb-power…

Now the fun part, how will having an LLM in my washing machine help anything

https://www.youtube.com/watch?v=LRq_SAuQDec

Re: AMD acquires Taalas to boost inference performance by etching models in silicon

#665

Post by therealpygon »

TechTechTech wrote: post #518

I think what will happen is what happened to something like 4K video decoding before where it ends up in silicon costing almost nothing to run extremely fast on device. "Good enough" LLM functionality (for the use case) will be on-die or on-chip for cars, appliances, etc. This will provide speeds of chatjimmy at a battery-level power consumption. Probably this will also happen for software engineering. Some usb-power…

I think this combined with a bit of memory and something like the “high-bandwidth flash” they just announced (if it works out), could be an interesting thing for some resident (burned) experts + active moe streamed from HBF.

I expect one day having small lower power demand drive sized devices with proprietary burned-in models that are quite fast running on-device in robotics and such. Commoditizing LLMs via burned and locked hardware seems likely when LLMs have stabilized (when we reach a year between releases again) and the hardware is capable and “disposable” enough. “Buy a robot and upgrade it forever* (5 years) with newer models (sold separately)”, at least until the planned hardware obsolescence that the “interface has changed to support newer hardware, so you’ll need to upgrade (again) to use the latest features”.

The plans basically write themselves.

Re: AMD acquires Taalas to boost inference performance by etching models in silicon

#666

Post by 5555watch »

Tade0 wrote: post #546
Zenst wrote: post #513

Earlier quoted context omitted.

With AI models like Mixture of Experts, many of those experts will be the real target here, as polished, refined and little to no change, they become fine candidates for being locked into silicon. Who knows, add some SRAM in there and small changes to those experts could be carried out without needing new silicon. Maybe AI models may become reduced to a collection of tiles you add to a chips one day, maybe sooner for…

An "Expert" is really just an unfortunate name for what amounts to a dense part of a sparse matrix and that's also an oversimplification. It doesn't actually specialise in anything in particular that one can point to. For this reason you can really transfer them between models.

Is it still true? I'd assume you should be able to freeze the matrix and unfreeze an expert block, before feeding particularly chosen training data. Or that doesn't work?

Re: AMD acquires Taalas to boost inference performance by etching models in silicon

#667

Post by empath75 »

mixermachine wrote: post #520
TechTechTech wrote: post #518

I think what will happen is what happened to something like 4K video decoding before where it ends up in silicon costing almost nothing to run extremely fast on device. "Good enough" LLM functionality (for the use case) will be on-die or on-chip for cars, appliances, etc. This will provide speeds of chatjimmy at a battery-level power consumption. Probably this will also happen for software engineering. Some usb-power…

Scaling a model on a chip is quite hard. ChatJimmy is based on Llama 3.1 8 billion. Kimi K3 has 2.8 trillion parameters. That are 350x more parameters. I would expect that Gemma 4 E2B (approx 5.1 billion parameters) or maybe even Gemma 4 26 billion A4B at some point is running on a chip.

You could imagine different layers on different chips, though, i think...

Re: AMD acquires Taalas to boost inference performance by etching models in silicon

#668

Post by viraptor »

andix wrote: post #241
walrus01 wrote: post #47

Earlier quoted context omitted.

I know it's a relatively tiny model, but damn, is that thing fast. It also mostly passes the "schlong" test https://pastes.io/YcxSi8Fp

It failed on my usual test. But it failed really fast: "A farmer has a wolf, a goat, and a cabbage. The wolf is imaginary and doesn't exist. He wants to cross the river, but the boat is only big enough to hold him and one of them. The farmer can't leave the wolf and the goat together, because the wolf will eat the goat. Similarly, he can't leave the goat and the cabbage together, because the goat will eat the cabbage…

I'm not sure one can fail this test. You can follow "wolf is not real", you can follow "wolf will eat the goat", or you can say the task is ambiguous. I could easily defend any of those.

Re: AMD acquires Taalas to boost inference performance by etching models in silicon

#669

Post by TeMPOraL »

tripzilch wrote: post #544
TeMPOraL wrote: post #474

Earlier quoted context omitted.

AI is already fast enough that human is a bottleneck. Hell, typing speed became a bottleneck like it was never before. I mean, if an agent can do half-decent work in less time than it takes the user to prompt them (and "user" in this context is a fast touch-typist like most programmers are), it's obvious it's not the agent that's the bottleneck anymore.

Because finishing someone else's (or something else's) "half decent work" to the point of "actually decent" becomes the bottleneck. This has always been the case for human project management, and LLMs just aren't at that level yet. It's more like everyone is speed running to how fast they can convince others that "half decent" is good enough. And for sure, newer models of LLM seem to be getting better at that .

> It's more like everyone is speed running to how fast they can convince others that "half decent" is good enough.

But that's what Agile is all about, isn't it? We've been speedrunning delivering increasingly smelly shit at increased velocity ever since SaaS became a thing, because ubiquitous Internet access is what allowed our industry to adopt the "lob feces over the fence for users to deal with" release model.

AI does speed that up, true (though since the market - and management - didn't catch up with it yet, we have a brief moment where we can use AI to increase quality while keeping usual delivery rate.)

Re: AMD acquires Taalas to boost inference performance by etching models in silicon

#670

Post by adityazero »

There are market verticals where this makes a lot of sense. Embedded systems and IoT devices comes to mind.

Even in data center space, I believe several layers can use fixed weights and remaining layers will compensate for the variations. Power savings will be huge so I think there is an incentive to do more of these.

Post reply on HN