I can actually see this being useful for fairly narrow workloads in dedicated devices where the model doesn't need to change very often and low-latency inference matters more than flexibility
I don't see it replacing general-purpose GPUs but it seems like a reasonable option for that kind of workload
AMD acquires Taalas to boost inference performance by etching models in silicon
581–590 of 710 posts
Re: AMD acquires Taalas to boost inference performance by etching models in silicon
#582This is a sign of the AI bubble bursting. The 2000 bubble also saw a flood of hardware accelerators from SSL to packet switching to firewalls to graphics, most of which failed.
Re: AMD acquires Taalas to boost inference performance by etching models in silicon
#583I think what will happen is what happened to something like 4K video decoding before where it ends up in silicon costing almost nothing to run extremely fast on device. "Good enough" LLM functionality (for the use case) will be on-die or on-chip for cars, appliances, etc. This will provide speeds of chatjimmy at a battery-level power consumption. Probably this will also happen for software engineering. Some usb-power…
Scaling a model on a chip is quite hard. ChatJimmy is based on Llama 3.1 8 billion. Kimi K3 has 2.8 trillion parameters. That are 350x more parameters. I would expect that Gemma 4 E2B (approx 5.1 billion parameters) or maybe even Gemma 4 26 billion A4B at some point is running on a chip.
I have found that for some personal prose-related projects, QWEN 3.6 35B A3B is an amazing model even quantized down to 4 bits. I actually find it's "writing" style as a GM for an LLM-powered solo text adventure game, better than even some of the faster/dummer frontier models like GPT-5.6-Luna or Haiku 4.5, and it runs (slowly) on a 3090 with a 80k context.
So I have faith in these embedded LLM chips when it comes to fun projects like that. I have not personally found my quantized QWEN good at agentic tasks, though, and it LOVES to make shit up when asking questions about documents in the prompt.
Re: AMD acquires Taalas to boost inference performance by etching models in silicon
#584Earlier quoted context omitted.
> Not sure if modern models "think" only by outputting blocks That's pretty much it - a small refinement to "Chain of Thought" prompting, where you tell the model explicitly in the prompt to "Think step by step" or similar, so it writes out more steps before giving a final answer, potentially catching some errors. The "thinking" models are tuned to do that without being prompted to, and to output the "thinking" marke…
My question was more about more complex problems, which no seem to be multi-turn somehow, or maybe just the harnesses make it look that way. I am curious what the drop in thoughput is for multi-turn answers, instead of one-shot. More in line with the current "agentic" use-cases.
Well at ≈15k tps, I think the bottleneck will not be the model (reasoning or tool calling), and attention will be shifted to the harness’ engineering again.
Probably, the usual initial suspects for “what makes computation slow” will become a focus point that needs to be optimized again: file access, network, etc.
Re: AMD acquires Taalas to boost inference performance by etching models in silicon
#585My understanding is that the chip carries the neural network itself while the weights are loaded on top, so any model that uses that same network topology should be loadable on demand.
They are loaded on top in the sense that they are contained in the upper layer(s) of the chip. So when you want to change the weights, you have to produce fewer masks for fabrication, which reduces cost and time to market.
Oh... That's not great - it'd be nice if it had a way to push updates without building a new chip.
OTOH, maybe because of this our future cyberdecks will have cartridge ports.
Re: AMD acquires Taalas to boost inference performance by etching models in silicon
#586I think what will happen is what happened to something like 4K video decoding before where it ends up in silicon costing almost nothing to run extremely fast on device. "Good enough" LLM functionality (for the use case) will be on-die or on-chip for cars, appliances, etc. This will provide speeds of chatjimmy at a battery-level power consumption. Probably this will also happen for software engineering. Some usb-power…
Scaling a model on a chip is quite hard. ChatJimmy is based on Llama 3.1 8 billion. Kimi K3 has 2.8 trillion parameters. That are 350x more parameters. I would expect that Gemma 4 E2B (approx 5.1 billion parameters) or maybe even Gemma 4 26 billion A4B at some point is running on a chip.
350x is only about 10-20 years of improvement, using CPU FLOPS as the benchmark.
Re: AMD acquires Taalas to boost inference performance by etching models in silicon
#587Earlier quoted context omitted.
Your examples worked on phones for over a decade. Maybe baking in a model that is "certified" to have some unconditioned truths + rest is pulled from external models/store could make sense. But AFAIK that doesn't exist and I'm not sure it can possibly be made. Perhaps society as a whole at least can work on an open corpus of training data, but I'm not holding my breath on this.
>Your examples worked on phones for over a decade. Nope. And not only not a decade ago, right now. If you have an Android or iPhone, you can give it clear and easy to understand instructions that Gemma 4 could complete[1] if it had tool calls on it, and that 100.00% of Claude, ChatGPT, Grok, Kimi, you name it, could understand and all complete if they had the access. The phones will fail to complete it. I just tried…
In my experience, they had a lot of stuff working well in the first few years they rolled out the home voice assistants - Alexa, google home, etc. But for whatever reason, they've spent the last eight(?) years silently breaking things that used to work. Stuff like audiobook playing, music alarms, or even messaging people.
Once they started seeing useful (if niche) functionality as a cost center, there wasn't really a world in which these could usefully exist. Their big bet now seems to be that LLMs will lead them to profitability - but whether that's from increased data harvesting, cheaper integrations, or because it'll be useful enough to charge subscription fees, I couldn't tell you.
Re: AMD acquires Taalas to boost inference performance by etching models in silicon
#588Earlier quoted context omitted.
Could the model or algorithm be changed to make it deterministic somehow? It could help a lot if there were reproduceable outputs from deterministic baked-in silicon.
You can make any LLM deterministic by dropping the temperature hyperparameter to zero. This will generally make them suck, though, a little bit of randomness is necessary for proper function.
You can also use a fixed seed for your prng. A hash of the input text (up to the current turn) should do.
Re: AMD acquires Taalas to boost inference performance by etching models in silicon
#589People are not talking enough how huge this is for robotics and IoT. Current robotics arhitectures are limited by tok/sec. How cares if its not a Fable model? This move undercuts NVIDIA directly.
no you're wrong. people are not talking about the load bearing seam that this strong decision has revealed towards veterinary care.
the implications for mental health of pet rats is huge.
Re: AMD acquires Taalas to boost inference performance by etching models in silicon
#590Earlier quoted context omitted.
Scaling a model on a chip is quite hard. ChatJimmy is based on Llama 3.1 8 billion. Kimi K3 has 2.8 trillion parameters. That are 350x more parameters. I would expect that Gemma 4 E2B (approx 5.1 billion parameters) or maybe even Gemma 4 26 billion A4B at some point is running on a chip.
It's quite telling that the 8B Taalas chip was already reticle-sized on TSMC N6. I mean, we're talking about a process that does ~100 MTr/mm², ROM needs about one transistor per bit, but can probably be packed more densely than general logic. Something like, say, 150 megabit/mm² is not a lot. N6 has a 850 mm² reticle limit. This roughly tracks, the article says the chip has 8B parameters and apparently spends about h…
A full wafer like Cerebras is about 60x that, and N2P has about 3x the transistor density. So right now it's technically feasible to etch a 1.4 trillion parameter model. So roughly DeepSeek-V4-Pro class. Imagine that running a factory, for example.