AMD acquires Taalas to boost inference performance by etching models in silicon

It's pretty clear if you see what's happening on current phones.

Autocorrect that works. Reply suggestions that almost work, just need to be tad more accurate (probably more of a data access issue than model) and a tad faster to look completely seamless. Screenshots with automated text detection and OCR and automatic interpretation (different suggested actions for when something on the picture looks like a web link, phone number, postal address, e-mail, or QR code, or an event poster). That's just a fraction of things I saw showing up on my Samsung phone over the last 6 months.

For over a year now, you could get a much better autocorrect and spell/grammar check, and a translator all in one, if you just pasted your text to a frontier model and asked it to check for errors or translate into target language. Now imagine being able to go through a round of such checks in a 1/100 of a second. You could have this running every keystroke, and suddenly the inline autocorrect/checks would not suck anymore.

Auto-linkifying that can correct for typos and doesn't need careful regex tuning because it understands from context what is meant to be a link or not. That's just one of many obvious things possible once you get local models running fast enough. Tip of an iceberg, and the first step to imagining all the other potential uses is to let go of the two mistaken beliefs people hold on to:

1. That LLMs are about written language. They're not; ever since "multimodal models" became a thing, tokenization extended to visual and audio space, and now textual and visual and aural inputs are all just regular, first-class tokens.

2. That chatting with the models is the only optimal way for end users to interact with AI. That's just artificially limiting yourself to the space of chat-based UI.

It will be, I'm just not happy with my blog post before making it live. The blog will have a live interactive chat and a link to the repo with the HDL.

I don't think anything I did was particularly novel, as I really just wanted to see how fast I could push a commodity FPGA to it's limit.

Scaling to an ASIC or getting into the billions of params is where the real engineering is! This was just a side project for a side project for me while the FPGA was idle

It’s described in this patent application [0]. There’s a bit of hand waving so the HC1 might be slightly different, but the gist is the same.

https://patents.justia.com/patent/20250123802

No I think you are thinking of Etched/Sohu.

Taalas' approach (at least for their demo'd product) is to bake the whole thing in completely. IIUC the optimiser can even see the weights while generating RTL. It's like there's an "uint8_t weights[] = " in the source code.

It makes fairly little sense for consumer hardware, unless you're embedding an absolutely tiny model. Since you're embedding the entire model in the chip itself, required chip size scales directly with model size. The reason they have llama3.1-8B as their model in https://chatjimmy.ai/ (their demo) is because it's very hard to do anything bigger than that, you literally start running out of physical space on the chip, and bigger chips cost money. You can't have a dishwasher using a chip the size of a B200 die to run a model you could run on a Raspberry Pi.

What's worse is that this is when it's already quantized to ~3 bits per parameter (which is fairly lobotomized). Yes, the chip will run it 1000X faster than the Raspberry Pi, but it will only be stupid faster.

Their press release explicitly lists that their HC1 puts the entire Llama 3.1 8B model on one 815 mm² TSMC N6 die, with about 53 billion transistors.

815 mm² is fucking huge. An RTX 5090 is 750 mm². A mid-to-high end consumer CPU die is something like 300 mm², with a lot of budget parts being significantly smaller than that, down to ~70 mm². Every square mm costs money.

If they upgraded from N6 to N3P they MIGHT be able to get as much as a ~35B class model on a a chip which fits in the reticle limit, maybe, probably not, because model weights aren't the only thing that needs to fit on the chip.

There are very serious issues with agentic performance in this setup, which is exactly where you would want something really fast. Their Llama 3.1 demo lists a context of 6,144, which is dramatically lower than the 131,072 Llama-3.1 supports.

Reasoning models are barely usable with contexts that short.

The reason for this is that to actually get those speeds, the KV cache needs to live in SRAM. You can't bake the KV cache into the circuitry since it... changes. They clearly don't have enough SRAM, and the problem gets worse the bigger you make the model since KV cache grows (sort of) with model dim. The longer you want to make your context, the more of your chip needs to be SRAM.

Frankly, I don't see the use-case for this tech. It's too expensive and too inflexible. Just doing what Cerebras did and making a wafer-sized chip which is mostly SRAM is a much better solution to serving LLMs at extreme speeds and you don't need to make a new chip every time a new model comes out.