Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

AMD acquires Taalas to boost inference performance by etching models in silicon

theregister.com

Post reply on HN
Print view View original

621–630 of 710 posts

Re: AMD acquires Taalas to boost inference performance by etching models in silicon

#621

Post by anigbrowl »

wmf wrote: post #317
anigbrowl wrote: post #267

Earlier quoted context omitted.

For fast Kimi K3? You're damn right I do

$1,000 only gets you the Qwen 27B cartridge. For Kimi K3 it would be more like $100,000 (and the "cartridge" is the size of a refrigerator).

Some hurried goalpost-moving there! If we take Qwen 27B as the benchmark of equivalent power (rather than its default configuration with no fine-tuning), I can still think of applications. Auto summarization for legal document databases and other high security/offline contexts, contraband detection, moderating websites like HN etc :)

Re: AMD acquires Taalas to boost inference performance by etching models in silicon

#622

Post by OneDeuxTriSeiGo »

orangeberrytea wrote: post #603
TechTechTech wrote: post #518

I think what will happen is what happened to something like 4K video decoding before where it ends up in silicon costing almost nothing to run extremely fast on device. "Good enough" LLM functionality (for the use case) will be on-die or on-chip for cars, appliances, etc. This will provide speeds of chatjimmy at a battery-level power consumption. Probably this will also happen for software engineering. Some usb-power…

Now the fun part, how will having an LLM in my washing machine help anything

Assistive technology.

Imagine an energy efficient IC for a small multimodal model that can do voice to text, text to speech, question/answer, tool calling, and structured output.

Wire that up to a microcontroller that parses the structured output to constrain the model (rather than giving the model direct hardware access).

Now you have an assistive tech mode for supporting vision impaired users without requiring them to configure an app on their phone, pair devices, etc. And so now the user can just speak to the washing machine to tell it what to do.

And because models are getting better and better at multi-language support, you can rely on a single model to cover a wide range of spoken languages. And therefore you don't need a bunch of variants of this chip for a single product line.

TLDR this gives a path to replace "always online" and "wifi enabled" devices with fully on-device capabilities without being forced to abandon assistive technology support.

Re: AMD acquires Taalas to boost inference performance by etching models in silicon

#623

Post by joseda-hg »

HPsquared wrote: post #615
orangeberrytea wrote: post #603

Earlier quoted context omitted.

Now the fun part, how will having an LLM in my washing machine help anything

It can ponder the meaning of its existence.

"My Job Is To Open and Close Doors" [1]

[1] - https://www.youtube.com/watch?v=49t-WWTx0RQ

Re: AMD acquires Taalas to boost inference performance by etching models in silicon

#624

Post by anigbrowl »

fwip wrote: post #333
anigbrowl wrote: post #306

Earlier quoted context omitted.

I see your argument but your example seems highly contrived. I can't think why you'd want to use something like this for something as dynamic as takeout ordering, where you might have to deal with bad customers, supply chain breakages, public health recalls, or any of many other probabilistic events. I think it's far more likely to see them used in safety critical applications where you need a capable model that can…

What safety critical applications would be a good fit for LLMs?

I'm not thinking of language models specifically, but large neural networks in silico. I feel like a 27B parameter model would likely be capable of flying and landing an airliner, for example.

Re: AMD acquires Taalas to boost inference performance by etching models in silicon

#625

Post by mossTechnician »

OneDeuxTriSeiGo wrote: post #622
orangeberrytea wrote: post #603

Earlier quoted context omitted.

Now the fun part, how will having an LLM in my washing machine help anything

Assistive technology. Imagine an energy efficient IC for a small multimodal model that can do voice to text, text to speech, question/answer, tool calling, and structured output. Wire that up to a microcontroller that parses the structured output to constrain the model (rather than giving the model direct hardware access). Now you have an assistive tech mode for supporting vision impaired users without requiring them…

I don't think that the ability to record and transcribe people's words will encourage manufacturers to remove Wi-Fi functionality. If anything, that would encourage them to keep it. Imagine the marketing possibilities.

Re: AMD acquires Taalas to boost inference performance by etching models in silicon

#626

Post by spicybright »

OneDeuxTriSeiGo wrote: post #622
orangeberrytea wrote: post #603

Earlier quoted context omitted.

Now the fun part, how will having an LLM in my washing machine help anything

Assistive technology. Imagine an energy efficient IC for a small multimodal model that can do voice to text, text to speech, question/answer, tool calling, and structured output. Wire that up to a microcontroller that parses the structured output to constrain the model (rather than giving the model direct hardware access). Now you have an assistive tech mode for supporting vision impaired users without requiring them…

that would be so cool, but it'll be a long time to make the cost of an AI chip cheaper than a wifi enabled one if that's even possible.

Re: AMD acquires Taalas to boost inference performance by etching models in silicon

#627

Post by pragma_x »

mNovak wrote: post #234

What I like about this, is that it significantly increases the probability of a sci-fi scenario where you're picking up a hot chip on the black market; rumor has it, Mythos 9 weights baked in...

What's old is new again. We'll be buying cartridges for AI capabilities on future architectures.

Re: AMD acquires Taalas to boost inference performance by etching models in silicon

#628

Post by imhoguy »

RobertDeNiro wrote: post #619
orangeberrytea wrote: post #603

Earlier quoted context omitted.

Now the fun part, how will having an LLM in my washing machine help anything

same way that having wifi does. by providing no actionable value, but boosting marketing materials

WiFi is fine for notification although Bluetooth would be enough to not get forgotten cloths stuck there for days.

But why it calls home and why we have to create accounts to just get a notification.

Re: AMD acquires Taalas to boost inference performance by etching models in silicon

#630

Post by Ericson2314 »

gf000 wrote: post #466
Ericson2314 wrote: post #461

Earlier quoted context omitted.

If your task has incremental rewards/feedback, you can push the "intelligence rate" simply by sampling the reward function faster. That's not fake, even if it not a substitute either. This is the "dumber but honest person that works harder" phenomenon, vs "lazy genius".

That's a good way to put it, but still my experience is that worse code bases are non-linearly harder to maintain and improve in the future, software tends to break down without a good enough base. Sure, in the future full rewrites and stuff like that will be just another "throw money at it" problem, but fundamentally software can get arbitrary complex and we barely know how to write large, maintainable code bases. N…

I totally agree with you on the first bit, but I also think that I am way better at deciding on how to refactor code bases than the LLM is.

Right now, I put models in low thinking mode during my refactors and hate waiting. I would much rather have a faster model that that maybe was slightly stupider, and I would wait far less long between prompts where it needs my valuable input.

Models that are dumb, but humble and fast, can be fine.

Post reply on HN