Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

AMD acquires Taalas to boost inference performance by etching models in silicon

theregister.com

Post reply on HN
Print view View original

521–530 of 712 posts

Re: AMD acquires Taalas to boost inference performance by etching models in silicon

#521

Post by dzhiurgis »

freekh wrote: post #423
anthonypasq wrote: post #74

Earlier quoted context omitted.

Personally I think Apple should have acquired them. if you could burn a gemma4 class model into an iphone and actually get extremely low latency and low battery usage it would feel like the future IMO. even if it means you wont get frontier intelligence, there might actually be incentive to buy a new mobile device every year again.

It would be cool if the future was a standard fairphone like module system where you could replace the model chip when you felt like it without having to shell out 1-2k $$$s for a new phone

Its wild but if chip is something like $30 and provides frontier intelligence then just throwing them away every 3 months isn't that big of a deal when a lot of us pay $50 to $150 to $1.5k per month on AI tools.

I don't think it needs to be on phone per-se. It can keep chugging in cloud - plenty of people use cheaper older models.

And I suspect the growth will slow eventually making taalas interations slower.

Re: AMD acquires Taalas to boost inference performance by etching models in silicon

#522

Post by akie »

threatripper wrote: post #421
linzhangrun wrote: post #269

Thinking that five or six years from now, Fable-level intelligence could be provided at 100x the current speed... makes me feel lost. I cannot imagine what the future will look like.

It's not just speed, it will consume a lot less energy per token, maybe even more than 100x difference. And cost for a chip that runs that one model will also go down a lot once volume scales up. They will end up way cheaper than flexible GPU chips. I expect AI models chopped up into building blocks where 99.9% of the compute is fixed but glued together with flexible "fine tuning" layers that will adapt them to speci…

Ohhh that's a very interesting hypothesis. I think you might be right.

Re: AMD acquires Taalas to boost inference performance by etching models in silicon

#523

Post by woadwarrior01 »

I suspect this acquisition was a response to NVIDIA's acquisition of Groq.

Cerebras is already public. AFAICT, there are 8 other startups in the space, some of which have mature products: Etched, d-Matrix, SambaNova, Tenstorrent, Positron, FuriosaAI, Rebellions and Fractile.

Re: AMD acquires Taalas to boost inference performance by etching models in silicon

#524

Post by azernik »

dyzone wrote: post #366
linzhangrun wrote: post #269

Thinking that five or six years from now, Fable-level intelligence could be provided at 100x the current speed... makes me feel lost. I cannot imagine what the future will look like.

It tells me that they have some kind of insider knowledge that the models have hit their limits and won't be getting much better, and it makes sense economically speaking to just bake the current models and use them for the next 5-10 years. Looks like we're near the top of the S curve.

No knowledge (insider or otherwise) is needed — could just be a bet, hedged against all the other bets a company AMD's size makes.

Re: AMD acquires Taalas to boost inference performance by etching models in silicon

#525

Post by heywoods »

thunderbird120 wrote: post #479
XCSme wrote: post #185

Earlier quoted context omitted.

I think this would make sense for consumer hardware, not for AI companies. AI companies constantly update/change stuff, new models come out, new requirements, etc. But if you ship an "ai-powered" dishwasher, it can come with the chip built-in to do computer vision and precisely target each spot, and will be sold as-is with no updates.

It makes fairly little sense for consumer hardware, unless you're embedding an absolutely tiny model. Since you're embedding the entire model in the chip itself, required chip size scales directly with model size. The reason they have llama3.1-8B as their model in https://chatjimmy.ai/ (their demo) is because it's very hard to do anything bigger than that, you literally start running out of physical space on the chip…

[dead]

Re: AMD acquires Taalas to boost inference performance by etching models in silicon

#526

Post by ben_w »

ronsor wrote: post #389
onion2k wrote: post #387

Earlier quoted context omitted.

Baking the base models on to ROM makes a lot of economic sense. Less so for consumers though, because it'd mean the phone is out of date in 3 months when a better model comes along.

It's a perfect reason to get consumers to buy a new phone every year again! They got bored of the camera.

Right now the models are doubling in performance (by the METR time horizon metric at least) every 4 months, so 3 doublings in a year; conversely, I hear (not my field) it takes around a year to make a prototype IC and another year to turn that into mass production, i.e. if the next (late-2026 model) iPhone has a chip like this, it will likely be with, at best, a late-2024 set of weights. I think you can get open-weights models today that have performance equivalent to the SOTA-late-2024 while fitting in the RAM of a (high end) 2025-26 phone.

At some point the music will stop on training bigger models, and when that happens it will make sense to have ROM weights (or 100% analog circuits given how noise-resistant LLMs are), but we'll know when that is because the investment bubble funding the training of new models will have burst.

Re: AMD acquires Taalas to boost inference performance by etching models in silicon

#529

Post by logicallee »

TeMPOraL wrote: post #487
IOT_Apprentice wrote: post #409

Earlier quoted context omitted.

I tried it. I asked where Bruce Lee was born. It stated he was born in Hong Kong. I challenged it and it went further naming a hospital there. I stated he was born in San Francisco and it apologized and then said his father was a missionary traveling in America, which was also wrong. Bruce’s father was a famous Cantonese Opera singer and actor. This model had zero information right, while being fast in responding. Un…

That's not what speed is useful for. I just pasted your comment and its whole inheritance chain to it, started my comment, and asked to generate a total of 9 completions, 3 from each of {current & next word, current paragraph, current paragraph + rewrite the entire paragraph}. Half of the answers were perfectly good (ironically, not the "next word" ones!), but the important bit, they came back near-instantly ("Genera…

>That's not what speed is useful for.

>I just pasted your comment and its whole inheritance chain to it,

Good idea. Only problem is it doesn't work. I just did the same thing with exactly this prompt:

>did the user IOT_Apprentice participate in the thread below and if, number and quote all of their comments. Only just number and quote the comments or write "Did not participate", do not add any commentary. Quote any comments by this user verbatim, exactly as input. Thread:

followed by pasting the thread[1]

And received the answer "IOT_Apprentice did not participate in the thread."[2] in 0.001s, even though they have literally the last comment in my quote and it's clearly legible.

It's particularly insidious because the understanding and thinking that is required to follow my requested answer format exactly is substantial - so based on the fact that it gets the format right and clearly understood the assignment, I would be inclined to believe that it would also be correct!

So to use your example, it's not just autocomplete, it's autocomplete that confidently returns "No matching results" in 0.001 seconds, even though there is a search term matching what you put in, right in the prompt itself that was sent to it. That is much worse than useless.

[1] prompt: https://ibb.co/CKVmRvtd

[2] result: https://ibb.co/BKdRKmyD

Re: AMD acquires Taalas to boost inference performance by etching models in silicon

#530

Post by tveita »

XCSme wrote: post #352
msdz wrote: post #334

Earlier quoted context omitted.

It’s not reasoning, the hardware demo uses a 3.-something generation Llama 8B. But it’s proven they can automate this (they didn’t etch eight billion weights by hand after all, obviously), so now the interesting question is whether they can scale it to more recent aka bigger models. After all, there’s already very useful models even for productivity at 27 or 35B.

My concern is that reasoning could involve some sequential steps that instant models don't. Not sure if modern models "think" only by outputting blocks, or there is a more complex mechanism at play.

> Not sure if modern models "think" only by outputting blocks

That's pretty much it - a small refinement to "Chain of Thought" prompting, where you tell the model explicitly in the prompt to "Think step by step" or similar, so it writes out more steps before giving a final answer, potentially catching some errors. The "thinking" models are tuned to do that without being prompted to, and to output the "thinking" markers around it, so they can be hidden from the user.

Post reply on HN