Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

AMD acquires Taalas to boost inference performance by etching models in silicon

theregister.com

Post reply on HN
Print view View original

501–510 of 712 posts

Re: AMD acquires Taalas to boost inference performance by etching models in silicon

#501

Post by amelius »

dave1010uk wrote: post #468

I'm surprised there's not more discussion about potential inflection points here. When technology gets faster, it opens up whole new classes of UX that were hard to predict For example, faster internet didn't mean being able to view 100x as many HTML4 web pages. It brought SaaS, streaming media and interactivity. I'm not good at predicting, but some ideas: 1. All information gets augmented in real time with personali…

I think the most important direction will be: you spawn hundreds of agents at the same time, and let them work in a breadth-first search style. So you will not necessarily get your answers much faster, but they will be much more thoroughly researched. And if you do want faster responses, you can do that at the expense of quality.

Re: AMD acquires Taalas to boost inference performance by etching models in silicon

#502

Post by darkwater »

dzhiurgis wrote: post #482
gf000 wrote: post #462

Earlier quoted context omitted.

It barely works even today, like Siri is laughably bad.

I mean the examples he gave definitely work. Mostly well I'd say as they are pretty primitive. What Siri is missing is more logical solutions and answers for recipes, etc (still suck even with chatgpt integration).

No, they don't work. Just asked Siri the other day "what's the weather tomorrow in $LOCATION" (where $LOCATION is a broader zone and not strictly a city) and the answer was the weather in a street called "$LOCATION Avenue" in a city 150km away.

Re: AMD acquires Taalas to boost inference performance by etching models in silicon

#503

Post by Cort3z »

I believe this is big news. Speculations here on out: I imagine this being baked into consumer products, greatly increasing the local token capability for consumers. They will suck the cloud-oriented companies' milkshake. Most users do not need extremely capable models, they just need some automation to do better web-searches, and get simple facts etc. If it can do simple coding tasks too, but at thousands of tokens per second, in stead of tens or hundreds, the development will benefit so much. It will benefit AMD in other ways too. I imagine they can start selling physical chips, usb-drive like devices, that just does llm. If you want a newer, better, model, you simply go to a store and buy one. Need more capability, buy more drives. Similar to physx back in the day, but with usb-c and a smaller footprint.

Re: AMD acquires Taalas to boost inference performance by etching models in silicon

#504

Post by tliltocatl »

amelius wrote: post #68
moshun wrote: post #60

Earlier quoted context omitted.

Considering the rate of model development and rail hopping, seems like baking models into silicon is speed-running obsolescence.

Not sure. You can fix the transistors but leave the connections between them open for flexibility, so you only need to change the manufacturing process for the upper masks for every new model.

I think they already do that, except it's not 1980 so you don't fix the upper mask, you fix the lowest metal layer (the upper layer is very coarse and is only useful for power). But even a single mask is still quite expensive.

Re: AMD acquires Taalas to boost inference performance by etching models in silicon

#505

Post by reqo »

One advantage that I see in models that are implemented as code is that they can quickly and cheaply be modified using LoRAs. What would the equivalent be in hardware? Another piece of hardware you would attach like adding a graphics card to a computer?

Re: AMD acquires Taalas to boost inference performance by etching models in silicon

#506

Post by overfeed »

dave1010uk wrote: post #468

I'm surprised there's not more discussion about potential inflection points here. When technology gets faster, it opens up whole new classes of UX that were hard to predict For example, faster internet didn't mean being able to view 100x as many HTML4 web pages. It brought SaaS, streaming media and interactivity. I'm not good at predicting, but some ideas: 1. All information gets augmented in real time with personali…

4. Pervasive , distributed dragnet surveillance under the misrepresentation that it's not a search until a human pulls the data. But a small on-device "E2E preserving" "safety" model that runs on your phone and snitches when illegal communication content is suspected.

Edit: also consider centralized Room-641A-type surveillance when models summarize and/or flag all calls processed by public telephony

Re: AMD acquires Taalas to boost inference performance by etching models in silicon

#507

Post by Philip-J-Fry »

planb wrote: post #431
LarsDu88 wrote: post #44

I'm surprised neither OpenAI nor Anthropic made this move first. The Chinese open weight models are pulling ahead and commoditizing their value proposition. Baking models onto silicon would've been the next logical move to get a moat. Google is already doing this and has an experimental project on top of already having TPUs and cramming their quantized flash onto individual TPUs for inference.

They are: https://openai.com/index/cerebras-partnership/ My guess is they only consider Luna "good enough" to justify the immense up-front investment to put it onto silicon, but Luna at 10x the current speed would be killer. If they're really pursuing live voice conversations with a hardware assistant, latency is more important than accuracy (for complex questions the assistant could always say something like "wait a…

Cerebras doesn't etch the model onto silicon though. They're basically just wafer scale GPUs. They're more flexible than etched silicon though because they can just run the next version of the model almost straight away.

Re: AMD acquires Taalas to boost inference performance by etching models in silicon

#508

Post by jgilias »

msteffen wrote: post #122

This is neat but IMO a little crazy. Something I personally haven’t seen much of, in all the discussions of model benchmarks and AI breakthroughs, is a distinction between “peak performance” and “reliable performance”. The “peak performance” of frontier models is very high: they’re solving open math problems, analyzing large codebases, etc. But my subjective impression is that “reliable performance” is mid at best: o…

What I’m doing is running a kind of a meta harness that uses different models (and underlying harnesses) to work on a problem, or review the solution. Idea is to get to an error rate better than each of the underlying models can provide. Same thing as sensor fusion.

Now, that’s slow and expensive although seems to work quite well (haven’t really evaled this properly, don’t have the time). If inference can be made fast and cheap, multi-model approaches like this would become more viable for more applications.

Re: AMD acquires Taalas to boost inference performance by etching models in silicon

#510

Post by A_D_E_P_T »

hypfer wrote: post #490
coderbants wrote: post #460

Earlier quoted context omitted.

I think the lifecycle for these chips could stretch far longer. If you're offering these models on a two year lifecycle, then you'd be able to stand up your top tier (wouldn't need to be frontier) at high speed. Run (for example) Kimi K3 on it and give it a brand name: AcmeAI Carbon Market it as your premier (only) model at high throughput. Two years later you stand up MSICs for the new state of the art with entirely…

I'm somewhat doubtful that we will be seeing something as large as Kimi K3 in silicon any time soon. This tech can definitely scale up from the current 8B prototype, but - at least as far as my limited understanding of the tech involved goes - you cannot just ASIC a trillion weights model due to physical size constraints. ___ Specification HC1 Model Llama 3.1 8B (hardwired) Process TSMC 6nm Die size 815mm² ___ So the…

Kimi K3 is huge, though. Deepseek V4 Flash is a much more moderate model (284B total), and it works extremely well. Models of that size, and smaller, are just going to keep getting better and better. Presumably there's a threshold below which models are not generally useful or competitive, but if models-on-silicon can scale up to just 256B, that would be really remarkable.
Post reply on HN