Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

How Taalas “prints” LLM onto a chip?

anuragk.com

Post reply on HN
Print view View original

81–90 of 266 posts

Re: How Taalas “prints” LLM onto a chip?

#82

Post by kioku »

I’m just wondering how this translates to computer manufacturers like Apple. Could we have these kinds of chips built directly into computers within three years? With insanely fast, local on-demand performance comparable to today’s models?

Re: How Taalas “prints” LLM onto a chip?

#83

Post by xattt »

kioku wrote: post #82

I’m just wondering how this translates to computer manufacturers like Apple. Could we have these kinds of chips built directly into computers within three years? With insanely fast, local on-demand performance comparable to today’s models?

Is it possible to supplement the model with a diff for updates on modular memory, or would severely impact perf?

Re: How Taalas “prints” LLM onto a chip?

#84

Post by ivan_gammel »

m101 wrote: post #71
ivan_gammel wrote: post #61

Earlier quoted context omitted.

It might be not that bad. “Good enough” open-weight models are almost there, the focus may shift to agentic workflows and effective prompting. The lifecycle of a model chip will be comparable to smartphones, getting longer and longer, with orchestration software being responsible for faster innovation cycles.

If you’re running at 17k tokens / s what is the point of multiple agents?

Different skills and context. Llama 3.1 8B has just 128k context length, so packing everything in it may be not a great idea. You may want one agent analyzing the requirements and designing architecture, one writing tests, another one writing implementation and the third one doing code review. With LLMs it’s also matters not just what you have in context, but also what is absent, so that model will not overthink it.

EDIT: just in case, I define agent as inference unit with specific preloaded context, in this case, at this speed they don’t have to be async - they may run in sequence in multiple iterations.

Re: How Taalas “prints” LLM onto a chip?

#85

Post by baq »

pfortuny wrote: post #37
thesz wrote: post #34

Earlier quoted context omitted.

800 mm2, about 90mm per side, if imagined as a square. Also, 250 W of power consumption. The form factor should be anything but thumbdrive.

mmmhhhhh 800mm2 ~= (30mm)2, which is more like a (biggish) thumb drive.

the radiator wouldn't be though

Re: How Taalas “prints” LLM onto a chip?

#86

Post by amelius »

thesz wrote: post #30

8B coefficients are packed into 53B transistors, 6.5 transistors per coefficient. Two-inputs NAND gate takes 4 transistors and register takes about the same. One coefficient gets processed (multiplied by and result added to a sum) with less than two two-inputs NAND gates. I think they used block quantization: one can enumerate all possible blocks for all (sorted) permutations of coefficients and for each layer place…

I'm looking forward to the model.toVHDL() method in PyTorch.

Re: How Taalas “prints” LLM onto a chip?

#87

Post by baq »

xattt wrote: post #83
kioku wrote: post #82

I’m just wondering how this translates to computer manufacturers like Apple. Could we have these kinds of chips built directly into computers within three years? With insanely fast, local on-demand performance comparable to today’s models?

Is it possible to supplement the model with a diff for updates on modular memory, or would severely impact perf?

this design at 7 transistors per weight is 99.9% burnt in the silicon forever.

Re: How Taalas “prints” LLM onto a chip?

#88

Post by spyder »

lm28469 wrote: post #50

Who's going to pay for custom chips when they shit out new models every two weeks and their deluded CEOs keep promising AGI in two release cycles?

It all depends on how cheap they can get. And another interesting thought: what if you could stack them? For example you have a base model module, then new ones come out that can work together with the old ones and expanding their capabilities.

Re: How Taalas “prints” LLM onto a chip?

#89

Post by Hendrikto »

amelius wrote: post #79
roncesvalles wrote: post #20

Earlier quoted context omitted.

That slot is called USB-C. I can fully imagine inference ASICs coming in powerbank form factor that you'd just plug and play.

> USB-C With these speeds you can run it over USB2, though maybe power is limiting.

USB-C is just a form factor and has nothing to do with which protocol you run at which speeds.

Re: How Taalas “prints” LLM onto a chip?

#90

Post by arisAlexis »

kioku wrote: post #82

I’m just wondering how this translates to computer manufacturers like Apple. Could we have these kinds of chips built directly into computers within three years? With insanely fast, local on-demand performance comparable to today’s models?

and run an outdated model for 3 years while progress is exponential? what is the point of that
Post reply on HN