Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

How Taalas “prints” LLM onto a chip?

anuragk.com

Post reply on HN
Print view View original

251–260 of 266 posts

Re: How Taalas “prints” LLM onto a chip?

#251

Post by p0u4a »

amelius wrote: post #86
thesz wrote: post #30

8B coefficients are packed into 53B transistors, 6.5 transistors per coefficient. Two-inputs NAND gate takes 4 transistors and register takes about the same. One coefficient gets processed (multiplied by and result added to a sum) with less than two two-inputs NAND gates. I think they used block quantization: one can enumerate all possible blocks for all (sorted) permutations of coefficients and for each layer place…

I'm looking forward to the model.toVHDL() method in PyTorch.

Pretty close to what you describe: https://github.com/fastmachinelearning/hls4ml

Re: How Taalas “prints” LLM onto a chip?

#252

Post by menaerus »

wmf wrote: post #206
menaerus wrote: post #45

Earlier quoted context omitted.

Not sure what you're on but I think what you said is incorrect. You can use hi-density HBM-enabled FPGA with (LP)DDR5 with sufficient number of logic elements to implement the inference. Reason why we don't see it in action is most likely in the fact that such FPGAs are insanely expensive and not so available off-the-shelf as the GPUs are.

Yeah, FPGA+HBM works but it has no advantage over GPU+HBM. If you want to store weights in FPGA LUTs/SRAM for insane speed you're going to need a lot of FPGAs because each one has very little capacity.

Ok, then I may have misunderstood what you were saying. If the only thing we are interested is to store all the weights into the block RAM or LUTs then, yeah, that wouldn't be possible. I understood the OPs question a bit differently too.

Re: How Taalas “prints” LLM onto a chip?

#253

Post by ivan_gammel »

wmf wrote: post #246
ivan_gammel wrote: post #245

Earlier quoted context omitted.

I feel weird defending Taalas here, but this argument is quite strange: of course it is more expensive now. It is irrelevant - all innovations are expensive at early stage. The question is, what this technology will cost tomorrow? Can it do for consumers what NPUs could not, offering good UX and quality of inference for reasonable price?

It will always be more expensive.

More expensive than what? How much equivalent low latency inference costs today?

I think you completely miss the UX point here. In 1997 CRT screens were mainstream, LCD was in the early stage, phones had antennas. In 2007 an iPhone with LCD touch screen changed the UX of computing forever. This tech that we see today is a precursor of technology that will dominate tomorrow. Today local inference is painful and expensive, it consumes a lot of energy. NPUs/GPUs solve nothing here, and they will always be less effective than hardwired models - by design. So only question is, when the consumer performance expectation for open-weight models will cross the price curve of specialized chips. It may happen earlier than for generic NPUs.

Re: How Taalas “prints” LLM onto a chip?

#254

Post by ivan_gammel »

bigyabai wrote: post #249
ivan_gammel wrote: post #244

Earlier quoted context omitted.

Most consumers do not care about reprogramming or fine-tuning and have no idea what NPU is. For many (including specifically those who still mourn dead AI companions, killed by 4o switch) the long term stability is much more important than benchmark performance of evergreen frontier model. If Taalas can produce a good hardwired model at scale at consumer market price point, a lot of people will just drop their AI sub…

> a lot of people will just drop their AI subscriptions. For a 2.5 kW Server? I don't see it happening, your money and electricity is better spent on CUDA compute.

>For a 2.5 kW Server?

I don’t see any reason why this should not drop to 100-300W at peak with maybe 100W*h of daily usage on smartphones.

Re: How Taalas “prints” LLM onto a chip?

#255

Post by ACCount37 »

ivan_gammel wrote: post #226
ACCount37 wrote: post #205

Earlier quoted context omitted.

Flash forward one year and it'll be H1 2026.

I don’t see why. Today frontier models are already 2 generations ahead of good enough. For many users they did not offer substantial improvement, sometimes things got even worse. What is going to happen within 1 year that will make users desire something beyond already working solution? LLMs are reaching maturity faster than smartphones, which now are good enough to stay on the same model for at least 5-6 years.

Any considerable bump in model capability craters my willingness to tolerate the ineptitude of less capable models. And I'm far from being alone in this.

Ever wondered why those stupid "they secretly nerfed the model!" myths persist? Why users report that "model got dumber", even if benchmarks stay consistent, even if you're on the inference side yourself and know with certainty that they are actually being served the same inference over the same exact weights on the same hardware quantized the same way?

Because user demands rise over time, always.

Users get a new flashy model, and it impresses them. It can do things the old model couldn't. Then they push it, and learn its limitations and quirks as they use it. And then it feels like it "got dumber" - because they got more aggressive about using it, got better at spotting all the ways it was always dumb in.

It's a treadmill, and you pretty much have to keep improving the models just to stay ahead of user expectations.

Re: How Taalas “prints” LLM onto a chip?

#256

Post by dzhiurgis »

fxnn wrote: post #217
yunohn wrote: post #181

Earlier quoted context omitted.

But the BTC mining algorithm has not and will not change. That’s the only reason ASICs atleast make a bit of sense for crypto. AI being static weights is already challenged with the frequent model updates we already see - but may even be a relic once we find a new architecture.

We can expect the model landscape to consolidate some day. Progress will become slower, innovations will become smaller. Not tomorrow, not next year, but the time will come. And then it'll increasingly make sense to build such a chip into laptops, smartphones, wearables. Not for high-end tasks, but to drive the everyday bread-and-butter tasks.

TPU usb-c dongle is less than $100 (widely used for detecting people in home assistant / frigate nvr camera feeds). If one-off $100 purchase can replace (and improve 10x by speed) anthropic subscription even for 12 months - I don't see why not.

Re: How Taalas “prints” LLM onto a chip?

#257

Post by dzhiurgis »

bigyabai wrote: post #195
hamdingers wrote: post #174

Earlier quoted context omitted.

This is a ridiculous mindset. Llama 3.1 8B can do lots of things today and it'll still be able to do those things tomorrow. If you baked one of these into a smart speaker that could call tools to control lights and play music, it will still be able to do that when Llama 4 or 5 or 6 comes out.

If you pay $1,500 for a Mistral ASIC that is beaten by a $15 Qwen ASIC that comes out six months later, you'd be feeling pretty dang ridiculous.

I'd pay for $100 chip that replaces anthropic sub and works 10x faster, even for 12 months.

Edit: assuming model owners will let this happen, which they wont

Re: How Taalas “prints” LLM onto a chip?

#258

Post by throwaway85825 »

ramraj07 wrote: post #131
throwaway85825 wrote: post #128

Few customers value tokens anywhere near what it costs the big API vendors. When the bubble pops the only survivors will be whoever can offer tokens at as close to zero cost as possible. Also whoever is selling hardware for local AI.

To those who use AI to get real work done in real products we build, we very much appreciate the value of each token given how much operational overhead it offsets. A bubble pop, if one does indeed happen, would at best be as disruptive as the dot-com bust.

It's a full employment program for security engineers.

How disruptive dot com was depends on where you were.

Re: How Taalas “prints” LLM onto a chip?

#259

Post by layla5alive »

MarsIronPI wrote: post #187
adeelk93 wrote: post #150

Earlier quoted context omitted.

And computers used to be the size of a room. I think they can get it to iPhone size in the future, this is an early prototype.

Well, there's a limit to how small we can make transistors with our current technology. As I understand it, Intel is already running into those limits with their new CPUs (they had to redesign the fins IIRC). I can imagine that without an actual breakthrough in chip manufacturing the size could stay large. That's not to say that a breakthrough won't happen, though.

Yes, in 2D, but NAND has been using layers for a while. We call HBM interposers 2.5D. 3D breakthrough would be pretty easy but for those pesky problems like power delivery and cooling. (/s)

But give that time (e.g. microfluidics) - something interesting is that it would be extra hard to use all layers at once, but NN might be a good fit, imagining that computation will be sparse (subsets activating simultaneously)...

Re: How Taalas “prints” LLM onto a chip?

#260

Post by layla5alive »

XorNot wrote: post #24
roncesvalles wrote: post #20

Earlier quoted context omitted.

That slot is called USB-C. I can fully imagine inference ASICs coming in powerbank form factor that you'd just plug and play.

Pretty sure it'd just be a thumbdrive. Are the Taalas chips particularly large in surface area?

Yes, bigger than a 5090's GB202 ASIC! :)
Post reply on HN