Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The path to ubiquitous AI (17k tokens/sec)

taalas.com

Post reply on HN
Print view View original

371–380 of 471 posts

Re: The path to ubiquitous AI (17k tokens/sec)

#371

Post by therealdrag0 »

otabdeveloper4 wrote: post #353
smusamashah wrote: post #221

Earlier quoted context omitted.

With this speed, you can keep looping and generating code until it passes all tests. If you have tests. Generate lots of solutions and mix and match. This allows a new way to look at LLMs.

This doesn't work. The model outputs the most probable tokens. Running it again and asking for less probable tokens just results in the same but with more errors.

Do you not have experience with agents solving problems? They already successfully do this. They try different things until they get a solution.

Re: The path to ubiquitous AI (17k tokens/sec)

#372

Post by JKCalhoun »

empath75 wrote: post #231
adityashankar wrote: post #47

Earlier quoted context omitted.

This depends on how much better the models will get from now in, if Claude Opus 4.6 was transformed into one of these chips and ran at a hypothetical 17k tokens/second, I'm sure that would be astounding, this depends on how much better claude Opus 5 would be compared to the current generation

Even an O3 quality model at that speed would be incredible for a great many tasks. Not everything needs to be claude code. Imagine Apple fine tuning a mid tier reasoning model on personal assistant/MacOs/IOS sorts of tasks and burning a chip onto the mac studio motherboard. Could you run claude code on it? Probably not, would it be 1000x better than Siri? absolutely.

Yeah, waiting for Apple to cut a die that can do excellent local AI.

Re: The path to ubiquitous AI (17k tokens/sec)

#373

Post by matu3ba »

ACCount37 wrote: post #62
mips_avatar wrote: post #52

I think the thing that makes 8b sized models interesting is the ability to train unique custom domain knowledge intelligence and this is the opposite of that. Like if you could deploy any 8b sized model on it and be this fast that would be super interesting, but being stuck with llama3 8b isn't that interesting.

The "small model with unique custom domain knowledge" approach has a very low capability ceiling. Model intelligence is, in many ways, a function of model size. A small model tuned for a given domain is still crippled by being small. Some things don't benefit from general intelligence much. Sometimes a dumb narrow specialist really is all you need for your tasks. But building that small specialized model isn't easy o…

> Engineering isn't free, models tend to grow obsolete as the price/capability frontier advances, and AI specialists are less of a commodity than AI inference is. I'm inclined to bet against approaches like this on a principle.

This does not sound like it will simplify the training and data side, unless their or subsequent models can somehow be efficiently utilized for that. However, this development may lead to (open source) hardware and distributed system compilation, EDA tooling, bus system design, etc getting more deserved attention and funding. In turn, new hardware may lead to more training and data competition instead of the current NVIDIA model training monopoly market. So I think you're correct for ~5 years.

Re: The path to ubiquitous AI (17k tokens/sec)

#374

Post by pigpop »

aurareturn wrote: post #140
audunw wrote: post #91

Earlier quoted context omitted.

It doesn’t make any sense to think you need the whole server to run one model. It’s much more likely that each server runs 10 instances of the model 1. It doesn’t make sense in terms of architecture. It’s one chip. You can’t split one model over 10 identical hardwire chips 2. It doesn’t add up with their claims of better power efficiency. 2.4kW for one model would be really bad.

We are both wrong. First, it is likely one chip for llama 8B q3 with 1k context size. This could fit into around 3GB of SRAM which is about the theoretical maximum for TSMC N6 reticle limit. Second, their plan is to etch larger models across multiple connected chips. It’s physically impossible to run bigger models otherwise since 3GB SRAM is about the max you can have on an 850mm2 chip. followed by a frontier-class l…

Aren't they only using the SRAM for the KV cache? They mention that the hardwired weights have a very high density. They say about the ROM part:

> We have got this scheme for the mask ROM recall fabric – the hard-wired part – where we can store four bits away and do the multiply related to it – everything – with a single transistor. So the density is basically insane.

I'm not a hardware guy but they seem to be making a strong distinction between the techniques they're using for the weights vs KV cache

> In the current generation, our density is 8 billion parameters on the hard wired part of the chip., plus the SRAM to allow us to do KV caches, adaptations like fine tuning, and etc.

Re: The path to ubiquitous AI (17k tokens/sec)

#375

Post by kleiba »

vntok wrote: post #125
kleiba wrote: post #107

Earlier quoted context omitted.

Care to enlighten me?

Don't ask a small LLM about precise minutiae factual information. Alternatively, ask yourself how plausible it sounds that all the facts in the world could be compressed into 8k parameters while remaining intact and fine-grained. If your answer is that it sounds pretty impossible... well it is.

Did you see the part in my original post where it said "Not unexpected for an 8k model"?

Re: The path to ubiquitous AI (17k tokens/sec)

#376

Post by jameslk »

The implications for RLM is really interesting. RLM is expensive because of token economics. But when tokens are so cheap and fast to generate, context size of the model matters a lot less

Also interesting implications for optimization-driven frameworks like DSPy. If you have an eval loop and useful reward function, you can iterate to the best possible response every time and ignore the cost of each attempt

Re: The path to ubiquitous AI (17k tokens/sec)

#377

Post by pigpop »

mike_hearn wrote: post #321
Alifatisk wrote: post #302

What's happening in the comment section? How come so many cannot understand that his is running Llama 3.1 8B? Why are people judging its accuracy? It's almost a 2 years old 8B param model, why are people expecting to see Opus level response!? The focus here should be on the custom hardware they are producing and its performance, that is whats impressive. Imagine putting GLM-5 on this, that'd be insane. This reminds m…

They don't give cost figures in their blog post but they do here: https://www.nextplatform.com/wp-content/uploads/2026/02/taal... Probably they don't know what the market will bear and want to do some exploratory pricing, hence the "contact us" API access form. That's fair enough. But they're claiming orders of magnitude cost reduction. > Is there really a potential market for hardware designed for one model only? I'…

Yes, there are all kinds of fuzzy NLP tasks that this would be great for. Jobs where you can chunk the text into small units and add instructions and only need a short response. You could burn through huge data sets very quickly using these chips.

Re: The path to ubiquitous AI (17k tokens/sec)

#378

Post by jameslk »

dust42 wrote: post #21

This is not a general purpose chip but specialized for high speed, low latency inference with small context. But it is potentially a lot cheaper than Nvidia for those purposes. Tech summary: - 15k tok/sec on 8B dense 3bit quant (llama 3.1) - limited KV cache - 880mm^2 die, TSMC 6nm, 53B transistors - presumably 200W per chip - 20x cheaper to produce - 10x less energy per token for inference - max context size: flexib…

> Certainly interesting for very low latency applications which need I’m really curious if context will really matter if using methods like Recursive Language Models[0]. That method is suited to break down a huge amount of context into smaller subagents recursively, each working on a symbolic subset of the prompt.

The challenge with RLM seemed like it burned through a ton of tokens to trade for more accuracy. If tokens are cheap, RLM seems like it could be beneficial here to provide much more accuracy over large contexts despite what the underlying model can handle

0. https://arxiv.org/abs/2512.24601

Post reply on HN