The demo: https://chatjimmy.ai/
OMFG this thing is fast.
its fast but try to get it to give you pi to 50 decimal places. it didnt go well for me.
AMD acquires Taalas to boost inference performance by etching models in silicon
81–90 of 712 posts
Re: AMD acquires Taalas to boost inference performance by etching models in silicon
#82I'm surprised neither OpenAI nor Anthropic made this move first. The Chinese open weight models are pulling ahead and commoditizing their value proposition. Baking models onto silicon would've been the next logical move to get a moat. Google is already doing this and has an experimental project on top of already having TPUs and cramming their quantized flash onto individual TPUs for inference.
Personally I think Apple should have acquired them. if you could burn a gemma4 class model into an iphone and actually get extremely low latency and low battery usage it would feel like the future IMO. even if it means you wont get frontier intelligence, there might actually be incentive to buy a new mobile device every year again.
That's actually a really good point... There's currently zero incentive to buying more hardware, and that's one very good reason do have a new one.
Re: AMD acquires Taalas to boost inference performance by etching models in silicon
#83Honestly, this is starting to make more and more sense. SOTA models are starting to converge to certain architecture and capabilities. I wouldn’t be surprised we end up with a base model ASIC + “fine tune” card where it’s a physical LoRA style adapter.
Imagine a multi-modal model with 1000's of tokens per second. Realtime inference for a host of applications. This is a BIG deal and will change the landscape in unfathomable ways. The https://chatjimmy.ai demo was impressive. Once models settle down this makes sense. Imagine a cartridge with a physical model on it. You purchase a cartridge and stick it in your computer/phone/server. Want to upgrade? By a new 'cartrid…
very interesting idea. i didnt think of that. i was just assuming youd have an additional one of these in your phone for actual lightning fast local inference
Re: AMD acquires Taalas to boost inference performance by etching models in silicon
#84Earlier quoted context omitted.
Llama 3.1 8B model
So this demo is around 90 times faster than typical speeds for the same model at openrouter, and around 30 times faster than the absolute fastest option available (Groq).
Re: AMD acquires Taalas to boost inference performance by etching models in silicon
#85I'm surprised neither OpenAI nor Anthropic made this move first. The Chinese open weight models are pulling ahead and commoditizing their value proposition. Baking models onto silicon would've been the next logical move to get a moat. Google is already doing this and has an experimental project on top of already having TPUs and cramming their quantized flash onto individual TPUs for inference.
A model can't be updated, and a chip that is only relevant for 6 months at max?
Re: AMD acquires Taalas to boost inference performance by etching models in silicon
#86The demo: https://chatjimmy.ai/
I know it's a relatively tiny model, but damn, is that thing fast. It also mostly passes the "schlong" test https://pastes.io/YcxSi8Fp
I read the paste, it got the etymology wrong, no? Schlong comes from shlang (snake), not shlemp (is this even a word? I don't speak Yiddish but couldn't find it on Google).
Oxford also claim that its first recorded use was from the 60s, not the 20s; https://www.oed.com/dictionary/schlong_n?tl=true
Re: AMD acquires Taalas to boost inference performance by etching models in silicon
#87Earlier quoted context omitted.
OMFG this thing is fast.
its fast but try to get it to give you pi to 50 decimal places. it didnt go well for me.
I think the same exact model running on CPU-only and RAM, or a small GPU, would do about the same? It's quite an old model now and small, you could throw a GGUF into llama-server or something for a side by side comparison.
https://huggingface.co/meta-llama/Llama-3.1-8B
As I remember just about any english language model from mid 2024 and earlier didn't even do well if you asked it to count sequentially from 0 to 100, nevermind calculating stuff.
Re: AMD acquires Taalas to boost inference performance by etching models in silicon
#88With web search and tool call a decent current generation model at the speed of the chatjimmy could do a lot. People saying it would be out of date are missing the point. It’s not going to make much sense for frontier companies that’s chasing the SOTA. But for a lot of business use cases if someone can put GLM 5.2 and sell it as a box, it would make so much sense.
My partner has been asking for a “completely private” model for doing research and shifting through volumes of data that can’t leave the office and $$$ for the current hardware makes no sense. It would be an easy sell if someone walks in with a black box that contains “ChatGPT”.
Re: AMD acquires Taalas to boost inference performance by etching models in silicon
#89I'm surprised neither OpenAI nor Anthropic made this move first. The Chinese open weight models are pulling ahead and commoditizing their value proposition. Baking models onto silicon would've been the next logical move to get a moat. Google is already doing this and has an experimental project on top of already having TPUs and cramming their quantized flash onto individual TPUs for inference.
Isn’t that kind of useless for the stock? It sounds complicated, unlike having number of CPUs go up.
It’s like talking about anything else than Megapixels when everyone was convinced that megapixels must go up in certain periods of the smartphone boom.
Re: AMD acquires Taalas to boost inference performance by etching models in silicon
#90I'm surprised neither OpenAI nor Anthropic made this move first. The Chinese open weight models are pulling ahead and commoditizing their value proposition. Baking models onto silicon would've been the next logical move to get a moat. Google is already doing this and has an experimental project on top of already having TPUs and cramming their quantized flash onto individual TPUs for inference.
It's a terrible moat. You etch the silicon then nobody wants to run it in 6 months because models have advanced that much further.
If a model is good enough today, it's still gonna be good enough in a year. Except you'll be able to serve it 1/100 of the price. Or 100x the speed.