Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

AMD acquires Taalas to boost inference performance by etching models in silicon

theregister.com

Post reply on HN
Print view View original

531–540 of 712 posts

Re: AMD acquires Taalas to boost inference performance by etching models in silicon

#531

Post by tiborsaas »

pastel8739 wrote: post #363
QuiDortDine wrote: post #251

Earlier quoted context omitted.

Did you use chatjimmy? It's somewhat terrifying to use when you think of the potential results with a better model. Ok, real life example: I now spend most of my time, as a developer, waiting for the agent to do its thing (after careful prompting, I'm also thinking about work stuff, don't worry I'm not useless). What if it gave back the same excellent results, but instantaneously? Why, then, I certainly would become…

This is the same pitch that people make about AI today. Speed isn’t the differentiator, quality is

Speed will be one of the killer features once you get closer to instant speeds of 300ms. Just remember what changes were made possible simply by upgrading from ADSL to broadband.

If inference speed goes up, I can launch the same query 5 times, evaluate the best result and proceed from there. Of course, evaluation is also instant, so in seconds I can get a near perfect solution. Or maybe 10 and I can pick what I like the best.

Re: AMD acquires Taalas to boost inference performance by etching models in silicon

#532

Post by logicallee »

v9v wrote: post #494
logicallee wrote: post #478

Earlier quoted context omitted.

if it's baked into silicon how can you two get different answers?

It still works the same way other LLMs do, by outputting the probability distribution over the possible completions (The weather is ... (sunny (50%), cloudy (50%))). Then the next token is sampled from this probability distribution (in our example the next word could be "sunny" or "cloudy" equally likely), which can result in different outputs every run.

Could the model or algorithm be changed to make it deterministic somehow? It could help a lot if there were reproduceable outputs from deterministic baked-in silicon.

Re: AMD acquires Taalas to boost inference performance by etching models in silicon

#533

Post by tiborsaas »

andix wrote: post #246
zxspectrum1982 wrote: post #150

Earlier quoted context omitted.

I'd gladly pay for a Claude Opus 4.6 Thinking High in silicon and use it for 1-2 years. It's good enough for many coding tasks.

With thousands of token per second output it would be an enormous waste of resources. Such chips are clearly made to process thousands of conversations simultaneously. Not necessarily in parallel. All LLM workflows are turn based right now, there are often seconds between turns until tool calls finish or users type the next message. If the LLM response only takes a few milliseconds, the chip can process hundreds of o…

With those speeds I can benchmark a batch of different approaches, compare the results and serve the results all within a second. It's quite amazing.

Re: AMD acquires Taalas to boost inference performance by etching models in silicon

#534

Post by XCSme »

thunderbird120 wrote: post #479
XCSme wrote: post #185

Earlier quoted context omitted.

I think this would make sense for consumer hardware, not for AI companies. AI companies constantly update/change stuff, new models come out, new requirements, etc. But if you ship an "ai-powered" dishwasher, it can come with the chip built-in to do computer vision and precisely target each spot, and will be sold as-is with no updates.

It makes fairly little sense for consumer hardware, unless you're embedding an absolutely tiny model. Since you're embedding the entire model in the chip itself, required chip size scales directly with model size. The reason they have llama3.1-8B as their model in https://chatjimmy.ai/ (their demo) is because it's very hard to do anything bigger than that, you literally start running out of physical space on the chip…

Yeah, makes sense, if it's good for very small models only, then there's no point, as those van already run on cheap consumer hardware.

Yet, maybe it can work well enough, so that as a manufacturer, you don't pay $50 for a PI, but only $0.50 for a tiny "hard-coded" chip.

The advantage can be that, as a LLM, as opposed to other types of chips, the use-cases could be more varied, so same chip could be use in different devices (robo vacuums, security cameras, ball-shooting training robots, etc.)

Re: AMD acquires Taalas to boost inference performance by etching models in silicon

#535

Post by petra »

thunderbird120 wrote: post #479
XCSme wrote: post #185

Earlier quoted context omitted.

I think this would make sense for consumer hardware, not for AI companies. AI companies constantly update/change stuff, new models come out, new requirements, etc. But if you ship an "ai-powered" dishwasher, it can come with the chip built-in to do computer vision and precisely target each spot, and will be sold as-is with no updates.

It makes fairly little sense for consumer hardware, unless you're embedding an absolutely tiny model. Since you're embedding the entire model in the chip itself, required chip size scales directly with model size. The reason they have llama3.1-8B as their model in https://chatjimmy.ai/ (their demo) is because it's very hard to do anything bigger than that, you literally start running out of physical space on the chip…

Maybe AMD will do a 3D memory layer for this chip for the kv cache. Possibly even something more dense than SRAM, like cgram, when it's ready.

Could this enable a reasonable context size ?

Re: AMD acquires Taalas to boost inference performance by etching models in silicon

#536

Post by askl »

boelboel wrote: post #198
Gigachad wrote: post #175

Earlier quoted context omitted.

Outside of this website I've never met a person who buys a new phone every year. It's closer to every 3-4 years for most people.

Closer to every 5-6 years these days and with ram prices going up it will be even longer. Especially with the low/mid range phones, which are most phones outside some developed countries, people will keep their phones as long as they can.

I mean are there even any reasons to buy a new phone?

If I compare the Pixel 6 Pro I'm using at the moment to current models, they are functionally identical. The only reason to upgrade might be getting a fresh battery and access to firmware updates.

Otherwise I'd be happy to continue using it for the next 10 years.

Re: AMD acquires Taalas to boost inference performance by etching models in silicon

#537

Post by dave1010uk »

overfeed wrote: post #506
dave1010uk wrote: post #468

I'm surprised there's not more discussion about potential inflection points here. When technology gets faster, it opens up whole new classes of UX that were hard to predict For example, faster internet didn't mean being able to view 100x as many HTML4 web pages. It brought SaaS, streaming media and interactivity. I'm not good at predicting, but some ideas: 1. All information gets augmented in real time with personali…

4. Pervasive , distributed dragnet surveillance under the misrepresentation that it's not a search until a human pulls the data. But a small on-device "E2E preserving" "safety" model that runs on your phone and snitches when illegal communication content is suspected. Edit: also consider centralized Room-641A-type surveillance when models summarize and/or flag all calls processed by public telephony

I hope this doesn't come to BCI.

Re: AMD acquires Taalas to boost inference performance by etching models in silicon

#538

Post by XCSme »

tveita wrote: post #530
XCSme wrote: post #352

Earlier quoted context omitted.

My concern is that reasoning could involve some sequential steps that instant models don't. Not sure if modern models "think" only by outputting blocks, or there is a more complex mechanism at play.

> Not sure if modern models "think" only by outputting blocks That's pretty much it - a small refinement to "Chain of Thought" prompting, where you tell the model explicitly in the prompt to "Think step by step" or similar, so it writes out more steps before giving a final answer, potentially catching some errors. The "thinking" models are tuned to do that without being prompted to, and to output the "thinking" marke…

My question was more about more complex problems, which no seem to be multi-turn somehow, or maybe just the harnesses make it look that way.

I am curious what the drop in thoughput is for multi-turn answers, instead of one-shot. More in line with the current "agentic" use-cases.

Re: AMD acquires Taalas to boost inference performance by etching models in silicon

#539

Post by amelius »

tliltocatl wrote: post #504
amelius wrote: post #68

Earlier quoted context omitted.

Not sure. You can fix the transistors but leave the connections between them open for flexibility, so you only need to change the manufacturing process for the upper masks for every new model.

I think they already do that, except it's not 1980 so you don't fix the upper mask, you fix the lowest metal layer (the upper layer is very coarse and is only useful for power). But even a single mask is still quite expensive.

But I suppose the interconnect masks don't have the resolution requirements of the masks for transistors. Therefore it could be a lot cheaper.

(Yes, you could fix a number of masks, e.g. entire logic gates, of course).

Re: AMD acquires Taalas to boost inference performance by etching models in silicon

#540

Post by askl »

wolttam wrote: post #77
LarsDu88 wrote: post #44

I'm surprised neither OpenAI nor Anthropic made this move first. The Chinese open weight models are pulling ahead and commoditizing their value proposition. Baking models onto silicon would've been the next logical move to get a moat. Google is already doing this and has an experimental project on top of already having TPUs and cramming their quantized flash onto individual TPUs for inference.

It's a terrible moat. You etch the silicon then nobody wants to run it in 6 months because models have advanced that much further.

Might be worse because it takes time between designing the silicon and having the first usable chips. So they're outdated the moment they hit the market or even before that.
Post reply on HN