Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

How Taalas “prints” LLM onto a chip?

anuragk.com

Post reply on HN
Print view View original

71–80 of 266 posts

Re: How Taalas “prints” LLM onto a chip?

#71

Post by m101 »

ivan_gammel wrote: post #61
m101 wrote: post #56

So if we assume this is the future, the useful life of many semiconductors will fall substantially. What part of the semiconductor supply chain would have pricing power in a world of producing many more different designs? Perhaps mask manufacturers?

It might be not that bad. “Good enough” open-weight models are almost there, the focus may shift to agentic workflows and effective prompting. The lifecycle of a model chip will be comparable to smartphones, getting longer and longer, with orchestration software being responsible for faster innovation cycles.

If you’re running at 17k tokens / s what is the point of multiple agents?

Re: How Taalas “prints” LLM onto a chip?

#72

Post by lm28469 »

lancebeet wrote: post #64
lm28469 wrote: post #50

Who's going to pay for custom chips when they shit out new models every two weeks and their deluded CEOs keep promising AGI in two release cycles?

You obviously don't believe that AGI is coming in two release cycles, and you also don't seem to have much faith in the new models containing massive improvements over the last ones. So the answer to who is going to pay for these custom chips seems to be you.

Why would I buy chips to run handicapped models when the 10+ llms players all offer free tier access to their 1t+ parameters models ?

Re: How Taalas “prints” LLM onto a chip?

#73

Post by ChrisMarshallNY »

dmurray wrote: post #31
XorNot wrote: post #24

Earlier quoted context omitted.

Pretty sure it'd just be a thumbdrive. Are the Taalas chips particularly large in surface area?

The only product they've announced at the moment [0] is a PCI-e card. It's more like a small power bank than a big thumb drive. But sure, the next generation could be much smaller. It doesn't require battery cells, (much) heat management, or ruggedization, all of which put hard limits on how much you can miniaturise power banks. [0] https://taalas.com/the-path-to-ubiquitous-ai/

I’m old enough to remember your typical computer filling warehouse-sized buildings.

Nowadays, your average cellphone has more computing power than those behemoths.

I have a micro SD card with 256GB capacity, and I think they are up to 2TB. On a device the size of a fingernail.

Re: How Taalas “prints” LLM onto a chip?

#75

Post by imtringued »

lm28469 wrote: post #59
brainless wrote: post #55

Earlier quoted context omitted.

New GPUs come out all the time. New phones come out (if you count all the manufacturers) all the time. We do not need to always buy the new one. Current open weight models < 20B are already capable of being useful. With even 1K tokens/second, they would change what it means to interact with them or for models to interact with the computer.

hm yeah I guess if they stick to shitty models it works out, I was talking about the models people use to actually do things instead of shitposting from openclaw and getting reminders about their next dentist appointment.

Considering that enamel regrowth is still experimental (only curodont exists as a commercial product), those dentist appointments are probably the most important routine healthcare appointments in your life. Pick something that is actually useless.

Re: How Taalas “prints” LLM onto a chip?

#76

Post by stavros »

ekianjo wrote: post #58
roncesvalles wrote: post #20

Earlier quoted context omitted.

That slot is called USB-C. I can fully imagine inference ASICs coming in powerbank form factor that you'd just plug and play.

Not if you need 200w power to run inference.

USB-C can do up to 240W. These days I power all my devices with a USB hub, even my Lipo charger.

Re: How Taalas “prints” LLM onto a chip?

#77

Post by imtringued »

moralestapia wrote: post #22

>HOW NVIDIA GPUs process stuff? (Inefficiency 101) Wow. Massively ignorant take. A modern GPUs is an amazing feat of engineering, particularly about making computation more efficient (low power/high throughput). Then proceeds to explain, wrongly, how inference is supposssedly implemented and draws conclusions from there ...

The way modern Nvidia GPUs perform inference is that they have a processor (tensor memory accelerator) that directly performs tensor memory operations which directly concedes that GPGPU as a paradigm is too inefficient for matrix multiplication.

Re: How Taalas “prints” LLM onto a chip?

#79

Post by amelius »

roncesvalles wrote: post #20
owenpalmer wrote: post #5

> Kinda like a CD-ROM/Game cartridge, or a printed book, it only holds one model and cannot be rewritten. Imagine a slot on your computer where you physically pop out and replace the chip with different models, sort of like a Nintendo DS.

That slot is called USB-C. I can fully imagine inference ASICs coming in powerbank form factor that you'd just plug and play.

> USB-C

With these speeds you can run it over USB2, though maybe power is limiting.

Re: How Taalas “prints” LLM onto a chip?

#80

Post by amelius »

wmf wrote: post #40
londons_explore wrote: post #18

So why only 30,000 tokens per second? If the chip is designed as the article says, they should be able to do 1 token per clock cycle... And whilst I'm sure the propagation time is long through all that logic, it should still be able to do tens of millions of tokens per second...

You still need to do a forward pass per token. With massive batching and full pipelining you might be able to break the dependencies and output one token per cycle but clearly they aren't doing that.

More aggressive pipelining will probably be the next step.
Post reply on HN