Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

How Taalas “prints” LLM onto a chip?

anuragk.com

Post reply on HN
Print view View original

51–60 of 266 posts

Re: How Taalas “prints” LLM onto a chip?

#51

Post by brainless »

roncesvalles wrote: post #19
byran wrote: post #3

This would be a very interesting future. I can imagine Gemma 5 Mini running locally on hardware, or a hard-coded "AI core" like an ALU or media processor that supports particular encoding mechanisms like H.264, AV1, etc. Other than the obvious costs (but Taalas seems to be bringing back the structured ASIC era so costs shouldn't be that low [1]), I'm curious why this isn't getting much attention from larger companies…

Well even programmable ASICs like Cerebras and Groq give many-multiples speedup over GPUs and the market has hardly reacted at all.

Seems both Nvidia (Groq) and OpenAI (Codex Spark) are now invested in the ASIC route one way or another.

Re: How Taalas “prints” LLM onto a chip?

#52

Post by brainless »

cpldcpu wrote: post #25

I wonder how well this works with MoE architectures? For dense LLMs, like llama-3.1-8B, you profit a lot from having all the weights available close to the actual multiply-accumulate hardware. With MoE, it is rather like a memory lookup. Instead of a 1:1 pairing of MACs to stored weights, you suddenly are forced to have a large memory block next to a small MAC block. And once this mismatch becomes large enough, there…

If each of the Expert models were etched in Silicon, it would still have massive speed boost, isn't it?

I feel printing ASIC is the main block here.

Re: How Taalas “prints” LLM onto a chip?

#53

Post by fuck_google »

fooker wrote: post #48
roncesvalles wrote: post #19

Earlier quoted context omitted.

Well even programmable ASICs like Cerebras and Groq give many-multiples speedup over GPUs and the market has hardly reacted at all.

> market has hardly reacted at all Guess who acqui-hired Groq to push this into GPUs? The name GPU has been an anachronism for a couple of years now.

[dead]

Re: How Taalas “prints” LLM onto a chip?

#54

Post by brainless »

If we can print ASIC at low cost, this will change how we work with models.

Models would be available as USB plug-in devices. A dense I hope lots of vendors will take note. Open weight models are abundant now. Even at a few thousand tokens/second, low buying cost and low operating cost, this is massive.

Re: How Taalas “prints” LLM onto a chip?

#55

Post by brainless »

lm28469 wrote: post #50

Who's going to pay for custom chips when they shit out new models every two weeks and their deluded CEOs keep promising AGI in two release cycles?

New GPUs come out all the time. New phones come out (if you count all the manufacturers) all the time. We do not need to always buy the new one.

Current open weight models < 20B are already capable of being useful. With even 1K tokens/second, they would change what it means to interact with them or for models to interact with the computer.

Re: How Taalas “prints” LLM onto a chip?

#56

Post by m101 »

So if we assume this is the future, the useful life of many semiconductors will fall substantially. What part of the semiconductor supply chain would have pricing power in a world of producing many more different designs?

Perhaps mask manufacturers?

Re: How Taalas “prints” LLM onto a chip?

#58

Post by ekianjo »

roncesvalles wrote: post #20
owenpalmer wrote: post #5

> Kinda like a CD-ROM/Game cartridge, or a printed book, it only holds one model and cannot be rewritten. Imagine a slot on your computer where you physically pop out and replace the chip with different models, sort of like a Nintendo DS.

That slot is called USB-C. I can fully imagine inference ASICs coming in powerbank form factor that you'd just plug and play.

Not if you need 200w power to run inference.

Re: How Taalas “prints” LLM onto a chip?

#59

Post by lm28469 »

brainless wrote: post #55
lm28469 wrote: post #50

Who's going to pay for custom chips when they shit out new models every two weeks and their deluded CEOs keep promising AGI in two release cycles?

New GPUs come out all the time. New phones come out (if you count all the manufacturers) all the time. We do not need to always buy the new one. Current open weight models < 20B are already capable of being useful. With even 1K tokens/second, they would change what it means to interact with them or for models to interact with the computer.

hm yeah I guess if they stick to shitty models it works out, I was talking about the models people use to actually do things instead of shitposting from openclaw and getting reminders about their next dentist appointment.

Re: How Taalas “prints” LLM onto a chip?

#60

Post by NinjaTrance »

lm28469 wrote: post #50

Who's going to pay for custom chips when they shit out new models every two weeks and their deluded CEOs keep promising AGI in two release cycles?

To run Llama 3.1 8B locally, you would need a GPU with a minimum of 16 GB of VRAM, such as an NVIDIA RTX 3090.

Talas promises a 10x higher throughtput, being 10x cheaper and using 10x less electricity.

Looks like a good value proposition.

Post reply on HN