Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

AMD acquires Taalas to boost inference performance by etching models in silicon

theregister.com

Post reply on HN
Print view View original

401–410 of 712 posts

Re: AMD acquires Taalas to boost inference performance by etching models in silicon

#401

Post by redox99 »

singingtoday wrote: post #310
redox99 wrote: post #144

Is there any LLM from exactly one year ago that would be worth running? In Aug 2025 you had - OpenAI o3 - Opus 4.1 - Gemini 2.5 Pro - Grok 4 Even if those were almost free to run, you'd be way better off with Deepseek flash 0731 or GPT 5.6 Luna, which already are almost free. Other than for things where the t/s are critical, it seems like a bad idea to etch a model into silicon.

We still run GPT 4.1 for some of our use cases. We want to replace it but are having trouble finding models that are as fast with similar or better intelligence.

There's nothing fast about GPT 4.1. It's ~50 tps AFAIK. Of course it doesn't use reasoning, but you can run modern models without thinking as well. GPT 5.6 Sol without reasoning should destroy it in intelligence.

Re: AMD acquires Taalas to boost inference performance by etching models in silicon

#402

Post by visarga »

QuiDortDine wrote: post #251
teaearlgraycold wrote: post #139

Earlier quoted context omitted.

My question is what changes about LLM use cases when you’re getting 1000 tok/s? Models in silicon might dramatically change how we think about them.

Did you use chatjimmy? It's somewhat terrifying to use when you think of the potential results with a better model. Ok, real life example: I now spend most of my time, as a developer, waiting for the agent to do its thing (after careful prompting, I'm also thinking about work stuff, don't worry I'm not useless). What if it gave back the same excellent results, but instantaneously? Why, then, I certainly would become…

AI helps you but also your competition, and gets factored in by investors while customers can use it to find better deals. The whole market is different even if a company did nothing.

Whatever you can cheaply do with AI is not a moat, if there is profit in there there will be quick imitation and competition will eat away those profits.

Models can be replaced easily, harnesses & AI tools too. And if cloud inference gets too expensive there are local models keeping the cloud prices hard capped.

Probably AI won't make anyone very rich.

Re: AMD acquires Taalas to boost inference performance by etching models in silicon

#403

Post by brailsafe »

nowittyusername wrote: post #328
desmaraisp wrote: post #316

Earlier quoted context omitted.

That's definitely super-enthousiast territory. Paying 80 bucks a month for AI is more than 99.99% of people would be willing to do

This will be considered very cheap within the year IMO. The value you get from AI is exponentially increasing and like all tech just takes some time to ramp up. Cell phones, internet and many other amenities when they came out many people were not willing to pay for but that all changed and considering how important AI tech is this will also be the case especially considering if its 100% private such as for that cart…

> The value you get from AI is exponentially increasing.

Perhaps in some cases, but the value I personally and professionally got out of LLMs reached a limit a while ago and has since kind of fluctuated between that limit and a bit less.

If the best model was instant, like the demo here, it could certainly provide more value, I guess, but I think the limit I'd quickly hit is the same one as now, which is how much of it do I want to produce, for what reasons?

Re: AMD acquires Taalas to boost inference performance by etching models in silicon

#404

Post by otabdeveloper4 »

onion2k wrote: post #387
chorizo wrote: post #257

Earlier quoted context omitted.

Baking the base models on to ROM makes a lot of economic sense. SRAM for the KV cache & fine-tunes, not so much. Sure you’d get incredible speeds but it’s not scalable from a die-size or cost perspective. Rather base model on ROM + KV cache on DRAM is much more scalable. Also this would work great for edge devices that have a 2-5 year lifecycle.

Baking the base models on to ROM makes a lot of economic sense. Less so for consumers though, because it'd mean the phone is out of date in 3 months when a better model comes along.

That's a software engineering problem. They just need to figure out how to fine-tune for alignment and tool usage.

That's the only thing the normie consumer cares for really.

Re: AMD acquires Taalas to boost inference performance by etching models in silicon

#405

Post by manmal »

spijdar wrote: post #179
RussianCow wrote: post #148

Earlier quoted context omitted.

That likely isn't as relevant for on-device iPhone usage as it is for Real Work™. I won't notice the difference between 50tps and 1000tps when asking Siri a question.

I don't know. As others have said, the Taalas chip wasn't small, or particularly low power, so it's hard to "imagine" what that tech in an cell phone chip might look like. But if the basic premise of "good enough LLM at insane throughput" holds, I think it could qualitatively change local uses of LLMs. At a certain speed point, you're able to move from request -> response to a cascade of tool calling and "subagents",…

> At a certain speed point, you're able to move from request -> response to a cascade of tool calling and "subagents"

That also needs server class hardware though. A phone won’t happily service the insane amount of IO, compute, and network that this cascade would require.

Re: AMD acquires Taalas to boost inference performance by etching models in silicon

#406

Post by hhh »

kzrdude wrote: post #390
AussieWog93 wrote: post #382

Earlier quoted context omitted.

Could we not just make bigger wafers, if the technology called for it?

The investment in bigger machines at the fab might set you back billions. I don't know about the lithography technology either, how easy you can scale it to larger wafers?

we seem to be in a phase of spending trillions on the computer, so while it isn’t likely, it isn’t impossible

Re: AMD acquires Taalas to boost inference performance by etching models in silicon

#407

Post by rf15 »

kzrdude wrote: post #390
AussieWog93 wrote: post #382

Earlier quoted context omitted.

Could we not just make bigger wafers, if the technology called for it?

The investment in bigger machines at the fab might set you back billions. I don't know about the lithography technology either, how easy you can scale it to larger wafers?

There's so much you'd need to scale, it's even more expensive than the humble few billions you spend on a new fab. And, beyond R&D, the scale itself also makes it significantly more expensive. This is an energetic dead end, you would need a country to fully commit to making this one thing.

Re: AMD acquires Taalas to boost inference performance by etching models in silicon

#408

Post by msteffen »

daishi55 wrote: post #159
msteffen wrote: post #122

This is neat but IMO a little crazy. Something I personally haven’t seen much of, in all the discussions of model benchmarks and AI breakthroughs, is a distinction between “peak performance” and “reliable performance”. The “peak performance” of frontier models is very high: they’re solving open math problems, analyzing large codebases, etc. But my subjective impression is that “reliable performance” is mid at best: o…

> out of 100 random questions I might think to ask, it’s likely to say something wrong or stupid a handful of times at least. What are some examples?

Well, I wrote my comment after an exchange similar to this, with the linked example chatbot (chatjimmy): https://pastebin.com/uEV7Pm8q

I asked it a slightly tricky math problem (I re-asked it the same problem to create the paste, and it did about as well the second time). It was unable to solve the problem, and it’s a small, old model, so…fair enough…but also its answer was pretty incoherent, with stuff like “Since A is an invariant set, it's always possible to find a cave that the fox cannot be in. Therefore, you can always catch the fox in that cave.” (…catch it…in the cave it can’t be in?).

Then, off the top of my head: Claude somewhat recently generated a Spark Job where the worker timeout was longer than the worker heartbeat, so workers would always inevitably be killed when they didn’t heartbeat within their timeout window. (also…neither option needed to be set?) Before I noticed the problem, I asked Claude why the job was taking so long, and it told me the data set was too large. More recently, there was a blog post by John Scalzi I was having a hard time finding, so I posed the problem to ChatGPT, and it came back with a blog post that didn’t include any version of the text I remembered and wasn’t really topically relevant (and maybe I hallucinated the blog post, but it could’ve said “I can’t find it” instead of “here you go”). On another occasion, I was trying to find a particular episode of Bob the Builder for my kids, so I Googled it, and Gemini kept giving me the wrong season and episode number, even after several rounds of “no, s5e6 is ‘that thing’, I’m looking for ‘this thing’.” Turned out the episode wasn’t on Amazon at all (which I had to tell it), and I had to go find it on YouTube.

That said, as I sit here scrolling through my history to see if I’ve forgotten any particularly good examples, I have to admit they do a better job than I’m giving them credit for. But I still wouldn’t have them write my email for me (the one time I tried that, when I was playing with openclaw, it sent a fairly demanding email to someone I didn’t know that well without asking for confirmation, and I had to go apologize and explain that I hadn’t really written the email, which was embarrassing), nor am I particularly excited to have chatjimmy as a permanent resident of my pocket.

Re: AMD acquires Taalas to boost inference performance by etching models in silicon

#409

Post by IOT_Apprentice »

QuiDortDine wrote: post #251
teaearlgraycold wrote: post #139

Earlier quoted context omitted.

My question is what changes about LLM use cases when you’re getting 1000 tok/s? Models in silicon might dramatically change how we think about them.

Did you use chatjimmy? It's somewhat terrifying to use when you think of the potential results with a better model. Ok, real life example: I now spend most of my time, as a developer, waiting for the agent to do its thing (after careful prompting, I'm also thinking about work stuff, don't worry I'm not useless). What if it gave back the same excellent results, but instantaneously? Why, then, I certainly would become…

I tried it. I asked where Bruce Lee was born. It stated he was born in Hong Kong. I challenged it and it went further naming a hospital there. I stated he was born in San Francisco and it apologized and then said his father was a missionary traveling in America, which was also wrong. Bruce’s father was a famous Cantonese Opera singer and actor.

This model had zero information right, while being fast in responding.

Unacceptable.

Re: AMD acquires Taalas to boost inference performance by etching models in silicon

#410

Post by Maxion »

mountainriver wrote: post #369
dyzone wrote: post #366

Earlier quoted context omitted.

It tells me that they have some kind of insider knowledge that the models have hit their limits and won't be getting much better, and it makes sense economically speaking to just bake the current models and use them for the next 5-10 years. Looks like we're near the top of the S curve.

What would possibly tell you that?

Expensive as fuck to make chips, only makes sense if you believe whatever model you're creating a chip out of will not become completely irrelevant in 5-10 years.
Post reply on HN