Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

AMD acquires Taalas to boost inference performance by etching models in silicon

theregister.com

Post reply on HN
Print view View original

391–400 of 710 posts

Re: AMD acquires Taalas to boost inference performance by etching models in silicon

#391

Post by manmal »

HDBaseT wrote: post #237
teaearlgraycold wrote: post #139

Earlier quoted context omitted.

My question is what changes about LLM use cases when you’re getting 1000 tok/s? Models in silicon might dramatically change how we think about them.

In the case on on-device/self-hosted LLMs. You ask your agent to implement xyz feature 10 times and use a model to compare the outputs and combine the best results. Raw intelligence becomes slightly less important when you can iterate and improve automatically. You can still claim it was "one shot" even when 30 different implementations were made then combined.

Problem is, there exists no judge model that will really pick the same winner that you would.

Re: AMD acquires Taalas to boost inference performance by etching models in silicon

#392

Post by Systemerror7A69 »

VBprogrammer wrote: post #56
wxw wrote: post #31

Earlier quoted context omitted.

I freakin' love this demo. It feels magical.

I had the same reaction but then I showed it to my partner. She completely didn't get it, in her words "how can it be thinking of a good answer when it's that quick?" I tried to explain but I fear were probably going to be adding artificial sleeps to these things to convince the masses it's doing something clever.

It's not thinking. Not in the way she probably meant. It can "think" that fast the same way a calculator can "think" that fast (kind of).

Because it's not human and not "thinking", it's a mathematical algorithm

Re: AMD acquires Taalas to boost inference performance by etching models in silicon

#393

Post by yassa9 »

pstuart wrote: post #384
encyclopedism wrote: post #22

Earlier quoted context omitted.

Imagine a multi-modal model with 1000's of tokens per second. Realtime inference for a host of applications. This is a BIG deal and will change the landscape in unfathomable ways. The https://chatjimmy.ai demo was impressive. Once models settle down this makes sense. Imagine a cartridge with a physical model on it. You purchase a cartridge and stick it in your computer/phone/server. Want to upgrade? By a new 'cartrid…

The cartridge could be a small mac-mini type unit connected and powered over thunderbolt. If it included like an m5 or m7 with 64GB of memory and a PCIe5/6 4TB Nvme it would be amazeballs. Hopefully when the bubble corrects and hardware advances and prices reset something like that will become available. Just even comparing compute from 10 years ago (Apple silicon vs Intel) and it's significant. 20 years it gets craz…

It is not linear anymore, take in consideration the Moore's law, the curve is nearly saturated now and gains in performance and memroy are not accelerating any more, BUT there is some hope with new different technologies, like the PHOTONIC chips , doing GEMMs through light particles instead of electrons

Re: AMD acquires Taalas to boost inference performance by etching models in silicon

#394

Post by manmal »

LarsDu88 wrote: post #375
teaearlgraycold wrote: post #139

Earlier quoted context omitted.

My question is what changes about LLM use cases when you’re getting 1000 tok/s? Models in silicon might dramatically change how we think about them.

Massive economic simulations with thousands if not millions of agents to front run the global economy and stock market. Fully interactive realtime NPCs in videogames at scale. Recommender systems that simulate individual consumers. Crazy shit

About your first example, isn’t the butterfly effect preventing this from being useful? One agent in your simulation decides to sell, and starts an avalanche, that won’t happen in reality?

Re: AMD acquires Taalas to boost inference performance by etching models in silicon

#395

Post by jaggederest »

pstuart wrote: post #370
jaggederest wrote: post #223

Earlier quoted context omitted.

They can't due to power density, I believe - they have to be run in a sandwiched waterblock with massive cooling, as far as I can tell. That's the biggest thing that baked weights gets you - a relatively modest watts-per-square-mm compare to cerebras, where they had to engineer a whole system to get the watts out of the chip

Do you think there's room for reducing power requirements? Obviously shrinking the process is a win, but is the existing implementation a "just make it work" phase that has opportunities to increase computational efficiency? With that kind of speed and if even lower power requirements, they could release mini compute units with USB4/Thunderbolt for plug and play inference.

I mean I think I read their spec sheet that it was 2.5kw on the asic but closer to 25kw with GPUs, for the same number of tokens (different latencies of course), but there's no way that a dedicated from-scratch matmul+memory chip can be less efficient than a GPU.

Re: AMD acquires Taalas to boost inference performance by etching models in silicon

#396

Post by 4k0hz »

XCSme wrote: post #352
msdz wrote: post #334

Earlier quoted context omitted.

It’s not reasoning, the hardware demo uses a 3.-something generation Llama 8B. But it’s proven they can automate this (they didn’t etch eight billion weights by hand after all, obviously), so now the interesting question is whether they can scale it to more recent aka bigger models. After all, there’s already very useful models even for productivity at 27 or 35B.

My concern is that reasoning could involve some sequential steps that instant models don't. Not sure if modern models "think" only by outputting blocks, or there is a more complex mechanism at play.

It's not really "instant", i.e. the text is still generated token-by-token, it's just super fast. Reasoning would work with this model without any changes to the chip but it's disabled for speed.

Re: AMD acquires Taalas to boost inference performance by etching models in silicon

#397

Post by chorizo »

onion2k wrote: post #387
chorizo wrote: post #257

Earlier quoted context omitted.

Baking the base models on to ROM makes a lot of economic sense. SRAM for the KV cache & fine-tunes, not so much. Sure you’d get incredible speeds but it’s not scalable from a die-size or cost perspective. Rather base model on ROM + KV cache on DRAM is much more scalable. Also this would work great for edge devices that have a 2-5 year lifecycle.

Baking the base models on to ROM makes a lot of economic sense. Less so for consumers though, because it'd mean the phone is out of date in 3 months when a better model comes along.

And the customers can wait for the new phone released next year. These are edge models - the average customer doesn’t need the latest frontier model. Just needs to be good enough for the features you promised.

Re: AMD acquires Taalas to boost inference performance by etching models in silicon

#398

Post by manmal »

dabbz wrote: post #233
retatop wrote: post #194

Earlier quoted context omitted.

But wouldn't higher tps allow for more reasoning or other hidden processes, potententially making a smarter model?

This is my thought as well. Models have to be intentional about which tokens they burn because there's a real lag time. If you can just fork out 10 different reasoning sessions at once with no regard for token waste/lag, you can compensate a smaller model with just doing more at once with it. No idea if this is reasonably true though.

I think that only works if you have checkpoints where all that reasoning can be checked against reality. Otherwise you get an army of armchair experts. LLMs are hilariously bad at home improvement advice btw, where reasoning alone won’t get you far.

Re: AMD acquires Taalas to boost inference performance by etching models in silicon

#399

Post by Certhas »

pastel8739 wrote: post #363
QuiDortDine wrote: post #251

Earlier quoted context omitted.

Did you use chatjimmy? It's somewhat terrifying to use when you think of the potential results with a better model. Ok, real life example: I now spend most of my time, as a developer, waiting for the agent to do its thing (after careful prompting, I'm also thinking about work stuff, don't worry I'm not useless). What if it gave back the same excellent results, but instantaneously? Why, then, I certainly would become…

This is the same pitch that people make about AI today. Speed isn’t the differentiator, quality is

So for every work produced by AI have ten separate agents review it thoroughly.

Re: AMD acquires Taalas to boost inference performance by etching models in silicon

#400

Post by aenis »

selcuka wrote: post #270
Melatonic wrote: post #134

Earlier quoted context omitted.

The Taalas chips are not physically small. And part of their secret (if you look at the design) is just locating a bunch of memory soldered on the edges ( I belive higher amounts of SRAM ? )

Their PoC chips are big, but then it's ridiculously fast (have you seen chatjimmy.ai?). Also they must be holding a bunch of patents.

Its a cool demo, but its gpt-3.5 level stupid, or worse.

edit: Ok, I will self-apologize. Its apparently a 3B model. Mighty impressive for what it does.

Post reply on HN