Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

AMD acquires Taalas to boost inference performance by etching models in silicon

theregister.com

Post reply on HN
Print view View original

631–640 of 712 posts

Re: AMD acquires Taalas to boost inference performance by etching models in silicon

#631

Post by OneDeuxTriSeiGo »

mossTechnician wrote: post #625
OneDeuxTriSeiGo wrote: post #622

Earlier quoted context omitted.

Assistive technology. Imagine an energy efficient IC for a small multimodal model that can do voice to text, text to speech, question/answer, tool calling, and structured output. Wire that up to a microcontroller that parses the structured output to constrain the model (rather than giving the model direct hardware access). Now you have an assistive tech mode for supporting vision impaired users without requiring them…

I don't think that the ability to record and transcribe people's words will encourage manufacturers to remove Wi-Fi functionality. If anything, that would encourage them to keep it. Imagine the marketing possibilities.

sure but once the capability is there it's not unlikely we'll see regulatory pressure from regions like the EU or certain US states to ban or heavily limit sale of "collect all your data" wifi enabled devices.

IMHO the only reason there's not been pushback from the EU already is because so much assistive tech is only possible with wireless connectivity currently. Once privacy conscious assistive tech is viable those crackdowns will probably be on the horizon even if only for the security risk associated with millions of internet connected devices with poor security posture.

Re: AMD acquires Taalas to boost inference performance by etching models in silicon

#633

Post by rnewme »

orangeberrytea wrote: post #603
TechTechTech wrote: post #518

I think what will happen is what happened to something like 4K video decoding before where it ends up in silicon costing almost nothing to run extremely fast on device. "Good enough" LLM functionality (for the use case) will be on-die or on-chip for cars, appliances, etc. This will provide speeds of chatjimmy at a battery-level power consumption. Probably this will also happen for software engineering. Some usb-power…

Now the fun part, how will having an LLM in my washing machine help anything

You load it, tell it what's in and it sets the program, tells you what it set and why. You approve and off it goes

Re: AMD acquires Taalas to boost inference performance by etching models in silicon

#634

Post by andrewvl »

downrightmike wrote: post #292
andrewvl wrote: post #70

It must be a “super model”. What will be if new model released? New chips?

Chip pops out like a gameboy cartridge. AI not working? Blow on it and jam it back in

Agree. Price is the question.

Re: AMD acquires Taalas to boost inference performance by etching models in silicon

#635

Post by walrus01 »

orangeberrytea wrote: post #603
TechTechTech wrote: post #518

I think what will happen is what happened to something like 4K video decoding before where it ends up in silicon costing almost nothing to run extremely fast on device. "Good enough" LLM functionality (for the use case) will be on-die or on-chip for cars, appliances, etc. This will provide speeds of chatjimmy at a battery-level power consumption. Probably this will also happen for software engineering. Some usb-power…

Now the fun part, how will having an LLM in my washing machine help anything

I'm sure the manufacturer will figure out a way to:

a) Have it display advertising on a screen on its front size (like some fridges now)

b) Only work with approved laundry pods

c) Other new revenue generating opportunities for its manufacturer

Re: AMD acquires Taalas to boost inference performance by etching models in silicon

#636

Post by dabbz »

I see a lot of discourse about it being fast-to-deprecation. But I see it a different way personally.

Modern LLMs are trying to do more with less. Focus on doing the right thing the first time. Even if we squeeze dumb LLMs, the significantly faster speed means quicker iterations. So a bad decision doesn't cost the time and inference costs that it cost before. It theoretically changes the scale of errant token spend.

I compare it to the 1 thousand monkeys on a typewriter. In this case it's 1,000 monkeys with stale training data of everything ever written and the ability to search the web.

Re: AMD acquires Taalas to boost inference performance by etching models in silicon

#637

Post by nomel »

try-working wrote: post #360
nomel wrote: post #170

Earlier quoted context omitted.

No, the point is inference speed and power.

you don't understand what I wrote.

I do. The point is inference speed and power, making previously impossible local inference possible. A side effect of that hardware optimization is fixed capabilities.

You've confused engineering compromise for malice, and reversed the purpose. For the model capabilities and inference power draw, what alternative do you see to a (at least mostly) fixed hardware model?

Re: AMD acquires Taalas to boost inference performance by etching models in silicon

#638

Post by pzo »

orangeberrytea wrote: post #603
TechTechTech wrote: post #518

I think what will happen is what happened to something like 4K video decoding before where it ends up in silicon costing almost nothing to run extremely fast on device. "Good enough" LLM functionality (for the use case) will be on-die or on-chip for cars, appliances, etc. This will provide speeds of chatjimmy at a battery-level power consumption. Probably this will also happen for software engineering. Some usb-power…

Now the fun part, how will having an LLM in my washing machine help anything

Washing machine was so inefficient whole time, it just heat water and spins. When you had to do by hand the hard decision is when to stop.

I believe there is a lot we can improve in washing machines

Re: AMD acquires Taalas to boost inference performance by etching models in silicon

#639

Post by lanyard-textile »

orangeberrytea wrote: post #603
TechTechTech wrote: post #518

I think what will happen is what happened to something like 4K video decoding before where it ends up in silicon costing almost nothing to run extremely fast on device. "Good enough" LLM functionality (for the use case) will be on-die or on-chip for cars, appliances, etc. This will provide speeds of chatjimmy at a battery-level power consumption. Probably this will also happen for software engineering. Some usb-power…

Now the fun part, how will having an LLM in my washing machine help anything

Think bigger: A washing machine and dryer with an arm and a camera.

It can sort your dirty and clean clothes before washing and after drying, based on your preferences.

It defaults to common sense things like washing similar materials and weights, keeping delicates in their own load, grouping sensitive colors like red.

Re: AMD acquires Taalas to boost inference performance by etching models in silicon

#640

Post by jrflo »

Tuna-Fish wrote: post #545
formerly_proven wrote: post #542

Earlier quoted context omitted.

It's quite telling that the 8B Taalas chip was already reticle-sized on TSMC N6. I mean, we're talking about a process that does ~100 MTr/mm², ROM needs about one transistor per bit, but can probably be packed more densely than general logic. Something like, say, 150 megabit/mm² is not a lot. N6 has a 850 mm² reticle limit. This roughly tracks, the article says the chip has 8B parameters and apparently spends about h…

Taalas exploits the low cardinality to store one 4-bit weight with one transistor. (They are using metal layer traces for the ROM, and connecting an access transistor to light up one of 16 options.) Their system is honestly very efficient for the weights, the problem is the KV-cache. That's why HC1 only supports such short context, they use SRAM for that and spend most of what's left of the die for it. The recent adv…

Do you really need a KV cache if inference is that fast though?
Post reply on HN