Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The path to ubiquitous AI (17k tokens/sec)

taalas.com

Post reply on HN
Print view View original

431–440 of 471 posts

Re: The path to ubiquitous AI (17k tokens/sec)

#431

Post by noisy_boy »

I'm curious how much of "hardcoding" is in the chip? Can it have parts that don't need changing much and "offload" the rest into some sort of high-speed/bandwidth interconnect?

Will we reach a state where we have chips on which models can be "flashed" like CPU firmware?

Or eventually will we reach a state where none of these tricks will be needed because like run-of-the mill Intel/AMD commodity CPUs, we will have full-power AI chips which will be part of an bigger/integrated mother-chip? Then what will happen to companies that do LLMs-as-a-service? Will they be forced to join and adapt becoming hybrid model+hardware shops?

I'm not knowledgeable enough about hardware but throwing these random ideas out in hopes of thought-provoking responses to learn from.

Re: The path to ubiquitous AI (17k tokens/sec)

#432

Post by test001only »

Alifatisk wrote: post #302

What's happening in the comment section? How come so many cannot understand that his is running Llama 3.1 8B? Why are people judging its accuracy? It's almost a 2 years old 8B param model, why are people expecting to see Opus level response!? The focus here should be on the custom hardware they are producing and its performance, that is whats impressive. Imagine putting GLM-5 on this, that'd be insane. This reminds m…

That is my concern too. A chip optimised for a model or specific model architecture will not be useful for long.

Re: The path to ubiquitous AI (17k tokens/sec)

#433

Post by smokel »

pankajdoharey wrote: post #417
dust42 wrote: post #21

This is not a general purpose chip but specialized for high speed, low latency inference with small context. But it is potentially a lot cheaper than Nvidia for those purposes. Tech summary: - 15k tok/sec on 8B dense 3bit quant (llama 3.1) - limited KV cache - 880mm^2 die, TSMC 6nm, 53B transistors - presumably 200W per chip - 20x cheaper to produce - 10x less energy per token for inference - max context size: flexib…

There is nothing new here. This has been demonstrated several times by previous researchers: https://arxiv.org/abs/2511.06174 https://arxiv.org/abs/2401.03868 For a real world use case, you would need an FPGA with terabytes of RAM. Perhaps it'll be a Off chip HBM. But for s large models, even that won't be enough. Then you would need to figure out NV-link like interconnect for these FPGAs. And we are back to square o…

This is new. You are citing FPGA prototypes. Those papers do not demonstrate the same class of scaling or hardware integration that Taalas is advocating. For one, the FPGA solutions typically use fixed multipliers (or lookup tables), the ASIC solution has more freedom to optimize routing for 4 bit multiplication.

Re: The path to ubiquitous AI (17k tokens/sec)

#435

Post by alexjplant »

dust42 wrote: post #21

This is not a general purpose chip but specialized for high speed, low latency inference with small context. But it is potentially a lot cheaper than Nvidia for those purposes. Tech summary: - 15k tok/sec on 8B dense 3bit quant (llama 3.1) - limited KV cache - 880mm^2 die, TSMC 6nm, 53B transistors - presumably 200W per chip - 20x cheaper to produce - 10x less energy per token for inference - max context size: flexib…

Most importantly this opens up an amazing future where we get the real version of the classic science fiction MacGuffin of a physical AI chip. Pair this with several TB of flash storage and you have persistent artificial consciousness that can be carried around with you. Bonus points if it's quirky, custom-trained and the chip is one of a kind that you stole from an evil corporation. Additional bonus points if the packaging is such that it's small enough to plug into the USB-C port on your smart glasses and has an eBPF module it can leverage to see what you're doing and talk to you in real time about your actions.

I enjoy envisioning futures more whimsical than "the bargain-basement LLM provider that my insurance company uses denied my claim because I chose badly-vectored words".

Re: The path to ubiquitous AI (17k tokens/sec)

#436

Post by doix »

I was wondering if/when this would happen. My friends and I would discuss this at the pub all the time, "LLM2RTL" or take it a step further and do the the whole process "LLM2GDS".

I couldn't find much info here, but I'm guessing they've built tooling to automatically convert model weights to RTL and the reason it's such an old model is that it takes a long time tape a chip out (especially the first one). Would be interesting to know how much is automated and how much is handcrafted.

I think the "next big thing" with AI hardware will be when they switch from "digital" implementations of LLMs to "analogue". We already know that we can lose some bits of precision and still have a "workable" model. If/when folks figure the fine-tuning out, I'm guessing it'll be another order of magnitude improvement.

Re: The path to ubiquitous AI (17k tokens/sec)

#437

Post by ahofmann »

test001only wrote: post #432
Alifatisk wrote: post #302

What's happening in the comment section? How come so many cannot understand that his is running Llama 3.1 8B? Why are people judging its accuracy? It's almost a 2 years old 8B param model, why are people expecting to see Opus level response!? The focus here should be on the custom hardware they are producing and its performance, that is whats impressive. Imagine putting GLM-5 on this, that'd be insane. This reminds m…

That is my concern too. A chip optimised for a model or specific model architecture will not be useful for long.

I just tried the demo and I think, this is huge! If they manage to build a chip in 2 or 3 years, that can run something like Opus 4.6 or even Sonnet, at that speed, the disruption in the world of software development will be more than we saw in the last 3-5 years. LLMs today are somewhat useful, but they are still too slow and expensive for a meaningful ralph loop. Being able to runs those loops (or if you want to call it "thinking") much faster, will enable a lot of stuff, that is not feasible today. Writing things like openclaw will not take weeks, but hours. Maybe even rewriting entire tools, kernels or OSes will be feasible because the LLM can run through almost endless tries.

Speed and cost wins over quality and this will also be true for LLMs.

Re: The path to ubiquitous AI (17k tokens/sec)

#438

Post by fragmede »

100721 wrote: post #114
small_model wrote: post #84

Earlier quoted context omitted.

Thats not how they work, pro-tip maybe don't comment until you have a good understanding?

Huh? Their words are an accurate, if simplified, description of how they work.

The simplification is where it loses granularity. I could describe every human's life as they were born and then they died. That's 100% accurate, but there's just a little something lost by simplifying that much.

Re: The path to ubiquitous AI (17k tokens/sec)

#439

Post by [deleted] »

akie wrote: post #427
Herring wrote: post #421

Earlier quoted context omitted.

Ok then I look forward to seeing DeepSeek running instantly at the end of April.

Why so negative lol. The speed and very reduced power use of this thing are nothing to be sneezed at. I mean, hardware accelerated LLMs are a huge step forward. But yeah, this is a proof of concept, basically. I wouldn't be surprised if the size factor and the power use go down even more, and that we'll start seeing stuff like this in all kinds of hardware. It's an enabler.

[deleted]

Re: The path to ubiquitous AI (17k tokens/sec)

#440

Post by Bolwin »

machiaweliczny wrote: post #179
dakolli wrote: post #12

try here, I hate llms but this is crazy fast. https://chatjimmy.ai/

We need that for this chinese 3B model that think 45s for hello world but also solves math.

Nanbeige. Yeah this seems ideal for models that scale test time compute
Post reply on HN