Earlier quoted context omitted.
Which model is it?
it said me it is llama 4 1.5B
never trust what a model says it is.
It tells me it is a variant of Codex.
AMD acquires Taalas to boost inference performance by etching models in silicon
511–520 of 712 posts
Re: AMD acquires Taalas to boost inference performance by etching models in silicon
#512I'm surprised there's not more discussion about potential inflection points here. When technology gets faster, it opens up whole new classes of UX that were hard to predict For example, faster internet didn't mean being able to view 100x as many HTML4 web pages. It brought SaaS, streaming media and interactivity. I'm not good at predicting, but some ideas: 1. All information gets augmented in real time with personali…
I think the most important direction will be: you spawn hundreds of agents at the same time, and let them work in a breadth-first search style. So you will not necessarily get your answers much faster, but they will be much more thoroughly researched. And if you do want faster responses, you can do that at the expense of quality.
You could probably do real-time deep research using that method, and than summarize and ask questions about the results.
That is a probably a significant jump in search quality for many queries, that people didn't take the time to research properly.
Re: AMD acquires Taalas to boost inference performance by etching models in silicon
#513People are not talking enough how huge this is for robotics and IoT. Current robotics arhitectures are limited by tok/sec. How cares if its not a Fable model? This move undercuts NVIDIA directly.
With AI models like Mixture of Experts, many of those experts will be the real target here, as polished, refined and little to no change, they become fine candidates for being locked into silicon. Who knows, add some SRAM in there and small changes to those experts could be carried out without needing new silicon.
Maybe AI models may become reduced to a collection of tiles you add to a chips one day, maybe sooner for some areas as you say, motor control for balance, vision systems, speach recognition systems etc, broken down, for robotoics, much is already there and just cost of battery/power holding much back.
Re: AMD acquires Taalas to boost inference performance by etching models in silicon
#514Earlier quoted context omitted.
It makes fairly little sense for consumer hardware, unless you're embedding an absolutely tiny model. Since you're embedding the entire model in the chip itself, required chip size scales directly with model size. The reason they have llama3.1-8B as their model in https://chatjimmy.ai/ (their demo) is because it's very hard to do anything bigger than that, you literally start running out of physical space on the chip…
And yield. Good luck getting good yield out of such ginormous chips. That said, I've been wondering if they could go with multiple smaller ones instead. Like one per layer maybe even? What are your thoughts on that? You seem to be more qualified than me on that matter.
Lots of chiplet interconnects hurt your max performance pretty severely and drives up your complexity a lot. There's a reason they want to keep it monolithic.
I already harped on Cerebras, but their approach of just copy/pasting a whole bunch of identical functional blocks, over-provisioning the chip by ~8%, and then just fusing off blocks with defects allows them to effectively have 100% yield on a wafer-sized monolithic chip. This is very desirable, and just another reason I like their approach better.
Re: AMD acquires Taalas to boost inference performance by etching models in silicon
#515everyone's worried about the model being obsolete before the chip ships. nobody's worried about the chip being obsolete before the model ships.
Re: AMD acquires Taalas to boost inference performance by etching models in silicon
#516[dead]
Re: AMD acquires Taalas to boost inference performance by etching models in silicon
#517People are not talking enough how huge this is for robotics and IoT. Current robotics arhitectures are limited by tok/sec. How cares if its not a Fable model? This move undercuts NVIDIA directly.
With AI models like Mixture of Experts, many of those experts will be the real target here, as polished, refined and little to no change, they become fine candidates for being locked into silicon. Who knows, add some SRAM in there and small changes to those experts could be carried out without needing new silicon. Maybe AI models may become reduced to a collection of tiles you add to a chips one day, maybe sooner for…
[dead]
Re: AMD acquires Taalas to boost inference performance by etching models in silicon
#518I think what will happen is what happened to something like 4K video decoding before where it ends up in silicon costing almost nothing to run extremely fast on device.
"Good enough" LLM functionality (for the use case) will be on-die or on-chip for cars, appliances, etc. This will provide speeds of chatjimmy at a battery-level power consumption.
Probably this will also happen for software engineering. Some usb-powered AI accelerator with Kimi K3 (and in future even better) performance running at 10K+ tokens/sec under 50W of power purchasable for almost no cost. Need a better model? Buy the new hardware. Old hardware is probably still fine for a lot of other use-cases. I expect China to be a big player here, it fits their open-model and hardware-manufacturing strategy.
Re: AMD acquires Taalas to boost inference performance by etching models in silicon
#519Yeah, so https://chatjimmy.ai/ ... the model is crap, but the speed is amazing. Worth checking out.
Worth wondering why they used a crap model.
Re: AMD acquires Taalas to boost inference performance by etching models in silicon
#520I think what will happen is what happened to something like 4K video decoding before where it ends up in silicon costing almost nothing to run extremely fast on device. "Good enough" LLM functionality (for the use case) will be on-die or on-chip for cars, appliances, etc. This will provide speeds of chatjimmy at a battery-level power consumption. Probably this will also happen for software engineering. Some usb-power…
Scaling a model on a chip is quite hard.
ChatJimmy is based on Llama 3.1 8 billion.
Kimi K3 has 2.8 trillion parameters.
That are 350x more parameters.
I would expect that Gemma 4 E2B (approx 5.1 billion parameters) or maybe even Gemma 4 26 billion A4B at some point is running on a chip.