Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

AMD acquires Taalas to boost inference performance by etching models in silicon

theregister.com

Post reply on HN
Print view View original

441–450 of 729 posts

Re: AMD acquires Taalas to boost inference performance by etching models in silicon

#441

Post by VBprogrammer »

Systemerror7A69 wrote: post #392
VBprogrammer wrote: post #56

Earlier quoted context omitted.

I had the same reaction but then I showed it to my partner. She completely didn't get it, in her words "how can it be thinking of a good answer when it's that quick?" I tried to explain but I fear were probably going to be adding artificial sleeps to these things to convince the masses it's doing something clever.

It's not thinking. Not in the way she probably meant. It can "think" that fast the same way a calculator can "think" that fast (kind of). Because it's not human and not "thinking", it's a mathematical algorithm

I know.

Re: AMD acquires Taalas to boost inference performance by etching models in silicon

#442

Post by miohtama »

ronsor wrote: post #389
onion2k wrote: post #387

Earlier quoted context omitted.

Baking the base models on to ROM makes a lot of economic sense. Less so for consumers though, because it'd mean the phone is out of date in 3 months when a better model comes along.

It's a perfect reason to get consumers to buy a new phone every year again! They got bored of the camera.

This stopped working some years ago.

Now you sell the same phone with higher price tag.

Re: AMD acquires Taalas to boost inference performance by etching models in silicon

#444

Post by petra »

chorizo wrote: post #257
Melatonic wrote: post #134

Earlier quoted context omitted.

The Taalas chips are not physically small. And part of their secret (if you look at the design) is just locating a bunch of memory soldered on the edges ( I belive higher amounts of SRAM ? )

Baking the base models on to ROM makes a lot of economic sense. SRAM for the KV cache & fine-tunes, not so much. Sure you’d get incredible speeds but it’s not scalable from a die-size or cost perspective. Rather base model on ROM + KV cache on DRAM is much more scalable. Also this would work great for edge devices that have a 2-5 year lifecycle.

AMD can let the SRAM be on a different chip. Maybe even something similar to their 3D cache. that could increase density to 20B[1]. They could also move from 6nm to 2nm. that would probably increase density by another 3x to 60B.

Add a bunch of chips together, and you get to a server that can run a 800B model, very fast and probably significantly cheaper than others.

[1]https://www.eetimes.com/taalas-specializes-to-extremes-for-e...

Re: AMD acquires Taalas to boost inference performance by etching models in silicon

#445

Post by niek_pas »

Maxion wrote: post #410
mountainriver wrote: post #369

Earlier quoted context omitted.

What would possibly tell you that?

Expensive as fuck to make chips, only makes sense if you believe whatever model you're creating a chip out of will not become completely irrelevant in 5-10 years.

There’s lots of use cases where the current models do fine though. A chip that can run a current cheap model at 100x would be amazing for things like detecting prohibited content on Facebook. You don’t need the 2030 equivalent of Fable for that, you need something that can cheaply process insane numbers of posts per day.

Re: AMD acquires Taalas to boost inference performance by etching models in silicon

#446

Post by Iolaum »

desmaraisp wrote: post #316
nowittyusername wrote: post #309

Earlier quoted context omitted.

Depends on how much it costs the consumer. If I could buy a "cartridge" of Kimi K3 for 300 bucks I 100% would buy that shit asap. Even if it's "no good" after lets say 4 months still would be worth it IMO.

That's definitely super-enthousiast territory. Paying 80 bucks a month for AI is more than 99.99% of people would be willing to do

Do think about b2b. Companies are already paying much more for AI. a new K3 (or similar model) every 6 months for a monthly rate of ~100$ per month is something MANY businesses would pay for. Then they could even sell them at half the price to consumers.

Re: AMD acquires Taalas to boost inference performance by etching models in silicon

#447

Post by IshKebab »

Maxion wrote: post #410
mountainriver wrote: post #369

Earlier quoted context omitted.

What would possibly tell you that?

Expensive as fuck to make chips, only makes sense if you believe whatever model you're creating a chip out of will not become completely irrelevant in 5-10 years.

No, it makes sense if whatever model you're creating a chip out of will not be irrelevant in 5 years even if it is much much faster & cheaper.

There are probably limited applications but not zero.

Re: AMD acquires Taalas to boost inference performance by etching models in silicon

#448

Post by rullelito »

NitpickLawyer wrote: post #343
bigyabai wrote: post #325

Earlier quoted context omitted.

Plug it in, and it's a old prototype with Gemma 5 weights baked onboard. Dammit, fucked by Craigslist again!

Back in the kazaa and limewire days, you'd sometimes try to get a movie / episode from a series, wait hours / days for it to download, and when it was done you had a ~50/50 chance to actually watch what you wanted or an old german porn movie :/

Better then vice versa!

Re: AMD acquires Taalas to boost inference performance by etching models in silicon

#449

Post by novaRom »

matheusmoreira wrote: post #425
linzhangrun wrote: post #269

Thinking that five or six years from now, Fable-level intelligence could be provided at 100x the current speed... makes me feel lost. I cannot imagine what the future will look like.

It makes me dizzy. I have no idea what is going to happen within even a year from now, can barely even imagine it. I'm trying to get the most out of it by redlining my AI subscriptions. Hopefully I'll manage to start a business in my niche. I don't even know if my niche will exist in the future.

Not just you. Hassabis in a recent interview looked a bit nervous when talking about AGI is rather sooner than many expected.

Re: AMD acquires Taalas to boost inference performance by etching models in silicon

#450

Post by trollbridge »

adgjlsfhk1 wrote: post #314
trebligdivad wrote: post #261

Earlier quoted context omitted.

What, even if it means you can run models without relying on the currently backlogged DRAM production?

The size of model we're talking about running doesn't need much if any dram.

The chatjimmy demo is using a model that needs 6-18GB of VRAM. That's not exactly trivial.

I could see it being feasible to get a Qwen-3.6-27b type of model done on something like this. Qwen-3.6-27b at 18tok/s would be a game changer.

Post reply on HN