Earlier quoted context omitted.
Wow, feels like Google web search in 1999.
If you still want the experience, go and browse McMaster Carr. Wizards designed that website.
I'm inspired by this website. It's incredible.
AMD acquires Taalas to boost inference performance by etching models in silicon
601–610 of 710 posts
Re: AMD acquires Taalas to boost inference performance by etching models in silicon
#602Earlier quoted context omitted.
It's a terrible moat. You etch the silicon then nobody wants to run it in 6 months because models have advanced that much further.
If a model is good enough today, it's still gonna be good enough in a year. Except you'll be able to serve it 1/100 of the price. Or 100x the speed.
I think we will eventually reach a point where this is the case, but at the moment it seems like you can throw virtually any non-trivial use-case at a model today and end up being more satisfied with the results that a model tomorrow gives.
I may just be closed minded as to what use-cases we have that current models are truly "good enough" (i.e. won't be dissatisfied when comparing results of today's model to tomorrow's model)
Re: AMD acquires Taalas to boost inference performance by etching models in silicon
#603I think what will happen is what happened to something like 4K video decoding before where it ends up in silicon costing almost nothing to run extremely fast on device. "Good enough" LLM functionality (for the use case) will be on-die or on-chip for cars, appliances, etc. This will provide speeds of chatjimmy at a battery-level power consumption. Probably this will also happen for software engineering. Some usb-power…
Now the fun part, how will having an LLM in my washing machine help anything
Re: AMD acquires Taalas to boost inference performance by etching models in silicon
#604Earlier quoted context omitted.
I find speed alone would be a game changer for current models. I hardly find any task anymore that the current frontier models can't do with max reasoning after several rounds of feedback (provided sufficient instruction and the right harness). But waiting an hour or more for reasoning to finish is getting really cumbersome. If they could do the same in seconds (and for cheap of course), I'm pretty sure we'd pretty s…
Can you give some examples of these tasks that require an hour or more of reasoning?
I've let it research and implement pretty complex stuff for my 3d engine. Like a realtime volumetric cloud rendering solution. The kind of stuff you can do yourself, but it would probably take weeks, if not months. I spent quite a bit discussing the design with it, knowing some details of how it should work from game dev conference talks. Once we agreed on a design and implementation plan with key steps and intermediaries, i just said "now implement it" and it happily designed algorithms, wrote and ran tests, generated and looked at screenshots and basically did everything you would expect from a really senior engine programmer for more than an hour of wall time. The end result was something that worked amazingly well after testing it myself and suggesting minor corrections that were my fault for not specifying them in the first place.
Re: AMD acquires Taalas to boost inference performance by etching models in silicon
#605I'm surprised there's not more discussion about potential inflection points here. When technology gets faster, it opens up whole new classes of UX that were hard to predict For example, faster internet didn't mean being able to view 100x as many HTML4 web pages. It brought SaaS, streaming media and interactivity. I'm not good at predicting, but some ideas: 1. All information gets augmented in real time with personali…
robotic probes on mars could run science experiments and report results on their own.
pacemakers could do deep analysis of heart signals and report problems.
Re: AMD acquires Taalas to boost inference performance by etching models in silicon
#606While the demo is incredible, I think that in most practical use-cases, models aren't very useful without tools (search, code execution, etc.).
Even if we assume reasoning latency drops to ~0ms (AFAIK this demo doesn't include reasoning at all), these use-cases will still remain relatively slow due to I/O of tool calls.
Re: AMD acquires Taalas to boost inference performance by etching models in silicon
#607Big if it pans out. Economic and financial ripple effects would be huge aside from the obvious: - reduction in electricity usage - OpenAI / Anthropic are dead in the water unless they start to license their models to fabs. - Every single one of those GPUs that all of those massive data centers contain become paperweights.
You still need the GPUs for training new models though right? This is only suitable for inference.
For sure, but I strongly doubt that we would need nearly as many.
Re: AMD acquires Taalas to boost inference performance by etching models in silicon
#608I think what will happen is what happened to something like 4K video decoding before where it ends up in silicon costing almost nothing to run extremely fast on device. "Good enough" LLM functionality (for the use case) will be on-die or on-chip for cars, appliances, etc. This will provide speeds of chatjimmy at a battery-level power consumption. Probably this will also happen for software engineering. Some usb-power…
Scaling a model on a chip is quite hard. ChatJimmy is based on Llama 3.1 8 billion. Kimi K3 has 2.8 trillion parameters. That are 350x more parameters. I would expect that Gemma 4 E2B (approx 5.1 billion parameters) or maybe even Gemma 4 26 billion A4B at some point is running on a chip.
There's certainly incentive to do so. And its only an engineering problem haha.
Re: AMD acquires Taalas to boost inference performance by etching models in silicon
#609Earlier quoted context omitted.
I'm somewhat doubtful that we will be seeing something as large as Kimi K3 in silicon any time soon. This tech can definitely scale up from the current 8B prototype, but - at least as far as my limited understanding of the tech involved goes - you cannot just ASIC a trillion weights model due to physical size constraints. ___ Specification HC1 Model Llama 3.1 8B (hardwired) Process TSMC 6nm Die size 815mm² ___ So the…
Kimi K3 is huge, though. Deepseek V4 Flash is a much more moderate model (284B total), and it works extremely well. Models of that size, and smaller, are just going to keep getting better and better. Presumably there's a threshold below which models are not generally useful or competitive, but if models-on-silicon can scale up to just 256B, that would be really remarkable.
My 2 cents to for your point:
- Deepseek V4 Flash is impressively capable. Sonnet still beats it out by a thin margin, but the real kicker is that a typical session with Sonnet at current API costs is ~$2. The same session with Deepseek is 2 cents (ha). Its even allowed me to consider offering free-with-limits API usage on my own app. - Taalas (or competitors) have a lot going for them. If anything I feel like they need to join hands with these smaller model makers and converge in 2028
Re: AMD acquires Taalas to boost inference performance by etching models in silicon
#610Compaction, busting the cache, and other issues like that will lose significance when you're running at 15k tokens per second like chatjimmy. Very interesting to think about what will change in the future.