Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

AMD acquires Taalas to boost inference performance by etching models in silicon

theregister.com

Post reply on HN
Print view View original

561–570 of 712 posts

Re: AMD acquires Taalas to boost inference performance by etching models in silicon

#561

Post by Closi »

mixermachine wrote: post #520
TechTechTech wrote: post #518

I think what will happen is what happened to something like 4K video decoding before where it ends up in silicon costing almost nothing to run extremely fast on device. "Good enough" LLM functionality (for the use case) will be on-die or on-chip for cars, appliances, etc. This will provide speeds of chatjimmy at a battery-level power consumption. Probably this will also happen for software engineering. Some usb-power…

Scaling a model on a chip is quite hard. ChatJimmy is based on Llama 3.1 8 billion. Kimi K3 has 2.8 trillion parameters. That are 350x more parameters. I would expect that Gemma 4 E2B (approx 5.1 billion parameters) or maybe even Gemma 4 26 billion A4B at some point is running on a chip.

Scaling is definetly hard - but there is no absolute requirement to put huge flagship models into this technology (although it might be possible over time).

A fairly dumb but FAST model has it's own totally distinct use-cases even if it can't be scaled in size. Think about a LLM-infused-Alexa where the response time is instant. Where you can request it looks at hotel options in Montreal, and it starts answering in half a second rather than a few minutes.

Plus some sort of slow smart + fast dumb combo architectures might also work really well for different classes of problems.

Re: AMD acquires Taalas to boost inference performance by etching models in silicon

#563

Post by harhargange »

Iolaum wrote: post #562

TBH Taalas was a company I was existed about as a consumer. A dense model like gemma4-31b or qwen3.6-27b running at 10k t/s sounds like an awesome thing to have. Would be willing to pay GPU prices for it.

My 7900XT runs it at 35 tokens per second.

Re: AMD acquires Taalas to boost inference performance by etching models in silicon

#564

Post by kllrnohj »

techpression wrote: post #417
kzrdude wrote: post #390

Earlier quoted context omitted.

The investment in bigger machines at the fab might set you back billions. I don't know about the lithography technology either, how easy you can scale it to larger wafers?

You also need to worry about yields, Apple, AMD etc can sell ”bad” chips as lower core versions, if you’re depending on whole wafer you have little room for error.

Well they still just do the same thing GPUs do where bad clusters are disabled. There's going to be a balance there for how big the clusters are and how many tiers they'll offer, but that's a known problem which can be well modeled.

Re: AMD acquires Taalas to boost inference performance by etching models in silicon

#566

Post by Tuna-Fish »

chorizo wrote: post #336
adrianN wrote: post #332

Earlier quoted context omitted.

It is my understanding that just baking the model itself into silicon only gives moderate gains because memory bandwidth remains a bottleneck.

The big benefit is ROM cells require fewer components than DRAM. So the chips would be tiny, dense, cheap and consume far less power.

It's not even just that. If you just built the rom chips separately and swapped them for the RAM of a normal accelerator, it would not help at all.

The trick is that every compute element in their system has it's own small pool of ROM, instead of putting all the ram behind a common pipe. ROM is just used because it's the densest kind of memory that can be fabricated on the same process as their logic.

Re: AMD acquires Taalas to boost inference performance by etching models in silicon

#567

Post by QuiDortDine »

bdangubic wrote: post #318
QuiDortDine wrote: post #251

Earlier quoted context omitted.

Did you use chatjimmy? It's somewhat terrifying to use when you think of the potential results with a better model. Ok, real life example: I now spend most of my time, as a developer, waiting for the agent to do its thing (after careful prompting, I'm also thinking about work stuff, don't worry I'm not useless). What if it gave back the same excellent results, but instantaneously? Why, then, I certainly would become…

> I now spend most of my time, as a developer, waiting for the agent to do its thing (after careful prompting, I'm also thinking about work stuff, don't worry I'm not useless). you need to launch 10-15 more terminals, who is waiting these days? :)

You sound like my boss! I'm not really into the whole "burnout" thing though.

Re: AMD acquires Taalas to boost inference performance by etching models in silicon

#568

Post by planb »

Philip-J-Fry wrote: post #507
planb wrote: post #431

Earlier quoted context omitted.

They are: https://openai.com/index/cerebras-partnership/ My guess is they only consider Luna "good enough" to justify the immense up-front investment to put it onto silicon, but Luna at 10x the current speed would be killer. If they're really pursuing live voice conversations with a hardware assistant, latency is more important than accuracy (for complex questions the assistant could always say something like "wait a…

Cerebras doesn't etch the model onto silicon though. They're basically just wafer scale GPUs. They're more flexible than etched silicon though because they can just run the next version of the model almost straight away.

Interesting. I did not know that. And still they got 15x speedup (https://www.cerebras.ai/blog/openai-gpt-oss-120b-runs-fastes...). I wonder how much additional speedup would be possible by really etching the model.

Re: AMD acquires Taalas to boost inference performance by etching models in silicon

#569

Post by Tuna-Fish »

logicallee wrote: post #532
v9v wrote: post #494

Earlier quoted context omitted.

It still works the same way other LLMs do, by outputting the probability distribution over the possible completions (The weather is ... (sunny (50%), cloudy (50%))). Then the next token is sampled from this probability distribution (in our example the next word could be "sunny" or "cloudy" equally likely), which can result in different outputs every run.

Could the model or algorithm be changed to make it deterministic somehow? It could help a lot if there were reproduceable outputs from deterministic baked-in silicon.

You can make any LLM deterministic by dropping the temperature hyperparameter to zero.

This will generally make them suck, though, a little bit of randomness is necessary for proper function.

Post reply on HN