Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

How Taalas “prints” LLM onto a chip?

anuragk.com

Post reply on HN
Print view View original

261–266 of 266 posts

Re: How Taalas “prints” LLM onto a chip?

#261

Post by layla5alive »

yonatan8070 wrote: post #111
dmurray wrote: post #31

Earlier quoted context omitted.

The only product they've announced at the moment [0] is a PCI-e card. It's more like a small power bank than a big thumb drive. But sure, the next generation could be much smaller. It doesn't require battery cells, (much) heat management, or ruggedization, all of which put hard limits on how much you can miniaturise power banks. [0] https://taalas.com/the-path-to-ubiquitous-ai/

I wouldn't call that size a small power bank. That chip is in the same ballpark as gaming GPUs, and based on the VRMs in the picture it probably draws about as much power. But as you said, the next generations are very likely to shrink (especially with them saying they want to do top of the line models in 2 generations), and with architecture improvements it could probably get much smaller.

Top of the line models will need more weights and more transistors, so the shrinking factors will be competing with growing factors, I'd expect them to keep maxing out the ASIC sizes to whatever is economically feasible.

Re: How Taalas “prints” LLM onto a chip?

#262

Post by fho »

kop316 wrote: post #230
fho wrote: post #224

Earlier quoted context omitted.

I think their "4-bit multiplier with a single transistor" bit is hinting at them using transistors in the sun-threshold regime.

So something that you can do with PDKs is add your own custom standard cell and tell the EDA tools to use them. This is actually pretty smart, this way you can use most of the foundry cells (which have been extensively validated) and focus on things like this "magic multiplier", that you will have to manually validate. This also makes porting across tech nodes easier if you manage only a handful of custom cells versu…

My "only" experience here is designing ASICs for Neuromorphic Chips. We used sub-threshold exclusively for linearity and energy reduction. No standard cells for us

Re: How Taalas “prints” LLM onto a chip?

#263

Post by ivan_gammel »

ACCount37 wrote: post #255
ivan_gammel wrote: post #226

Earlier quoted context omitted.

I don’t see why. Today frontier models are already 2 generations ahead of good enough. For many users they did not offer substantial improvement, sometimes things got even worse. What is going to happen within 1 year that will make users desire something beyond already working solution? LLMs are reaching maturity faster than smartphones, which now are good enough to stay on the same model for at least 5-6 years.

Any considerable bump in model capability craters my willingness to tolerate the ineptitude of less capable models. And I'm far from being alone in this. Ever wondered why those stupid "they secretly nerfed the model!" myths persist? Why users report that "model got dumber", even if benchmarks stay consistent, even if you're on the inference side yourself and know with certainty that they are actually being served th…

> users report that "model got dumber"

I have seen this with ChatGPT progression from 4o to 5.2 applied to the newest model. Old prompts stop working reliably, different hallucination modes etc.

Re: How Taalas “prints” LLM onto a chip?

#264

Post by dogma1138 »

jgalt212 wrote: post #134
dogma1138 wrote: post #113

Earlier quoted context omitted.

FPGAs would be less efficient than GPUs. FPGAs don’t scale if they did all GPUs would’ve been replaced by FPGAs for graphics a long time ago. You use an FPGA when spinning a custom ASIC doesn’t makes financial sense and generic processor such as a CPU or GPU is overkill. Arguably the middle ground here are TPUs, just taking the most efficient parts of a “GPU” when it comes to these workloads but still relying on memo…

I thought it was because the number logic elements in a GPU is orders of magnitude higher than in a FPGA, rather than just processing speed. And GPU processing is inherently parallel so the GPU beats the FPGA just based on transistor count.

With FPGA you are sacrificing performance for flexibility you are far less efficient in transistors for any given task than with a dedicated ASIC even if it’s a general compute ASIC like a GPU is today.

The reason no one is building large FPGAs is that there is no market for them.

If an H200 scale FPGA was viable we would have one.

Re: How Taalas “prints” LLM onto a chip?

#265

Post by yonatan8070 »

layla5alive wrote: post #261
yonatan8070 wrote: post #111

Earlier quoted context omitted.

I wouldn't call that size a small power bank. That chip is in the same ballpark as gaming GPUs, and based on the VRMs in the picture it probably draws about as much power. But as you said, the next generations are very likely to shrink (especially with them saying they want to do top of the line models in 2 generations), and with architecture improvements it could probably get much smaller.

Top of the line models will need more weights and more transistors, so the shrinking factors will be competing with growing factors, I'd expect them to keep maxing out the ASIC sizes to whatever is economically feasible.

Naturally they'll always have a big expensive SKU, but the existence of a Threadripper doesn't automatically obsolete the Ryzen 3

Re: How Taalas “prints” LLM onto a chip?

#266

Post by tmzt »

yonatan8070 wrote: post #112
NitpickLawyer wrote: post #68

Earlier quoted context omitted.

> Since it'll encourage Open weighs models Is this accurate? I don't know enough about hardware, but perhaps someone could clarify: how hard would it be to reverse engineer this to "leak" the model weights? Is it even possible? There are some labs that sell access to their models (mistral, cohere, etc) without having their models open. I could see a world where more companies can do this if this turns out to be a via…

It's likely possible to extract model weights from the chip's design, but you'd need tooling at the level of an Intel R&D lab, not something any hobbyist could afford. I doubt anyone would have the skills, wallet, and tools to RE one of these and extract model weights to run them on other hardware. Maybe state actors like the Chinese government or similar could pull that off.

Or a grinder and a camera. See CCC of years past.
Post reply on HN