Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

How Taalas “prints” LLM onto a chip?

anuragk.com

Post reply on HN
Print view View original

211–220 of 266 posts

Re: How Taalas “prints” LLM onto a chip?

#211

Post by yunwal »

throwthrowuknow wrote: post #168
darkwater wrote: post #157

Earlier quoted context omitted.

In a not-too-distant future (5 years?) small LLMs will be good enough to be used as generic models for most tasks. And if you have a dedicated ASIC small enough to fit in an iPhone, you have a truly local AI device with the bonus point that you get something really new to sell in every new generation (i.e. acces to an even more powerful model)

it doesn’t need to go in the phone if it only takes a few milliseconds to respond and is cheap

Perceptible latency is somewhere between 10 and 100ms. Even if an LLM was hosted in every aws region in the world, latency would likely be annoying if you were expecting near-realtime responses (for example, if you were using an llm as autocomplete while typing). If, say, apple had an LLM on a chip any app could use some SDK to access, it could feasibly unlock a whole bunch of usecases that would be impractical with a network call.

Also, offline access is still a necessity for many usecases. If you have something like an autocomplete feature that stops working when you're on the subway, the change in UX between offline and online makes the feature more disruptive than helpful.

https://www.cloudping.co/

Re: How Taalas “prints” LLM onto a chip?

#212

Post by darkwater »

bigyabai wrote: post #203
hamdingers wrote: post #198

Earlier quoted context omitted.

I'm equally capable of making up numbers to support my perspective but I don't see the point.

The point is that the GP's mindset is not very ridiculous if you value things by a price/utility ratio. Software and hardware advancements will lead to buyer's remorse faster than people get an ROI from local inference.

SW and HW advancements will bring this topic in the "good enough for vast majority" field, thus making GP point moot. You don't care if your LLM ASIC chip is not the latest one because it works for the use you purchased it for. The highly dynamical nature of LLM itself will make part of the advantage of upgradable software not that interesting anymorw. [1]

[1] although security might be a big enough reason for upgrades to still be required

Re: How Taalas “prints” LLM onto a chip?

#213

Post by yunohn »

dangus wrote: post #208
yunohn wrote: post #181

Earlier quoted context omitted.

But the BTC mining algorithm has not and will not change. That’s the only reason ASICs atleast make a bit of sense for crypto. AI being static weights is already challenged with the frequent model updates we already see - but may even be a relic once we find a new architecture.

Sounds to me like there’s potential to use these for established models to provide cost/scale advantage while frontier models will run in the existing setup.

IME llama et all require LoRA or fine-tuning to be usable. That's their real value vs closed source massive models, and their small size makes this possible, appealing, and doable on a recurring basis as things evolve. Again, rendering ASICs useless.

Re: How Taalas “prints” LLM onto a chip?

#215

Post by fxnn »

yunohn wrote: post #213
dangus wrote: post #208

Earlier quoted context omitted.

Sounds to me like there’s potential to use these for established models to provide cost/scale advantage while frontier models will run in the existing setup.

IME llama et all require LoRA or fine-tuning to be usable. That's their real value vs closed source massive models, and their small size makes this possible, appealing, and doable on a recurring basis as things evolve. Again, rendering ASICs useless.

Read the blog post. It mentions that their chip has a small SRAM which can store LoRA.

Re: How Taalas “prints” LLM onto a chip?

#216

Post by mikeurbach »

androiddrew wrote: post #125
amelius wrote: post #86

Earlier quoted context omitted.

I'm looking forward to the model.toVHDL() method in PyTorch.

Is this a thing?

I gave a short talk about compiling PyTorch to Verilog at Latte '22. Back then we were just looking at a simple dot product operation, but the approach could theoretically scale up to whole models.

https://capra.cs.cornell.edu/latte22/paper/2.pdf

https://www.youtube.com/watch?v=QxwZpYfD60g

Re: How Taalas “prints” LLM onto a chip?

#217

Post by fxnn »

yunohn wrote: post #181
RobotToaster wrote: post #118

Earlier quoted context omitted.

It reminds me of the switch from GPUs to ASICs in bitcoin mining. I've been expecting this to happen.

But the BTC mining algorithm has not and will not change. That’s the only reason ASICs atleast make a bit of sense for crypto. AI being static weights is already challenged with the frequent model updates we already see - but may even be a relic once we find a new architecture.

We can expect the model landscape to consolidate some day. Progress will become slower, innovations will become smaller. Not tomorrow, not next year, but the time will come.

And then it'll increasingly make sense to build such a chip into laptops, smartphones, wearables. Not for high-end tasks, but to drive the everyday bread-and-butter tasks.

Re: How Taalas “prints” LLM onto a chip?

#218

Post by throwup238 »

sowbug wrote: post #182
amelius wrote: post #86

Earlier quoted context omitted.

I'm looking forward to the model.toVHDL() method in PyTorch.

Ugh, quick, everyone start panic-buying FPGAs now.

largest FPGAs have on the order of tens of millions of logic cells/elements. They’re not even remotely big enough to emulate these designs except to validate small parts of it at a time and unlike memory chips or GPUs, companies don’t need millions of them to scale infrastructure.

(The chips also cost tens of thousands of dollars each)

Re: How Taalas “prints” LLM onto a chip?

#220

Post by slow_typist »

wmf wrote: post #207
darkwater wrote: post #157

Earlier quoted context omitted.

In a not-too-distant future (5 years?) small LLMs will be good enough to be used as generic models for most tasks. And if you have a dedicated ASIC small enough to fit in an iPhone, you have a truly local AI device with the bonus point that you get something really new to sell in every new generation (i.e. acces to an even more powerful model)

The Taalas approach is much more expensive than the NPU that phones already have.

Yes but not in five years. The chips will be dirt cheap by then. We‘ll get “intelligent” washing machines that will discuss the amount of detergent and eventually berate us. Toasters with voice input. And really annoying elevators. Also bugs that keep an extremely low RF profile (only phoning home when the target is talking business).
Post reply on HN