Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The path to ubiquitous AI (17k tokens/sec)

taalas.com

Post reply on HN
Print view View original

91–100 of 471 posts

Re: The path to ubiquitous AI (17k tokens/sec)

#91

Post by audunw »

aurareturn wrote: post #56
dust42 wrote: post #46

Earlier quoted context omitted.

> Don’t forget that the 8B model requires 10 of said chips to run. Are you sure about that? If true it would definitely make it look a lot less interesting.

Their 2.4 kW is for 10 chips it seems based on the next platform article. I assume they need all 10 chips for their 8B q3 model. Otherwise, they would have said so or they would have put a more impressive model as the demo. https://www.nextplatform.com/2026/02/19/taalas-etches-ai-mod...

It doesn’t make any sense to think you need the whole server to run one model. It’s much more likely that each server runs 10 instances of the model

1. It doesn’t make sense in terms of architecture. It’s one chip. You can’t split one model over 10 identical hardwire chips

2. It doesn’t add up with their claims of better power efficiency. 2.4kW for one model would be really bad.

Re: The path to ubiquitous AI (17k tokens/sec)

#92

Post by app13 »

trentnix wrote: post #89

The speed of the chatbot's response is startling when you're used to the simulated fast typing of ChatGPT and others. But the Llama 3.1 8B model Taalas uses predictably results in incorrect answers, hallucinations, poor reliability as a chatbot. What type of latency-sensitive applications are appropriate for a small-model, high-throughput solution like this? I presume this type of specialization is necessary for robo…

Routing in agent pipelines is another use. "Does user prompt A make sense with document type A?" If yes, continue, if no, escalate. That sort of thing

Re: The path to ubiquitous AI (17k tokens/sec)

#93

Post by Lionga »

taneq wrote: post #29
gozucito wrote: post #25

Can it scale to an 800 billion param model? 8B parameter models are too far behind the frontier to be useful to me for SWE work. Or is that the catch? Either way I am sure there will be some niche uses for it.

Spam. :P

so 90% of the AI market?

Re: The path to ubiquitous AI (17k tokens/sec)

#94

Post by japoneris »

I am super happy to see people working on hardware for local llm. Yet, isnt it premature ? Space is still evolving. Today, i refuse to buy a gpu because i do not know what will be the best model tomorrow. Waiting to get a on the shelf device to run an opus like model

Re: The path to ubiquitous AI (17k tokens/sec)

#95

Post by moralestapia »

audunw wrote: post #91
aurareturn wrote: post #56

Earlier quoted context omitted.

Their 2.4 kW is for 10 chips it seems based on the next platform article. I assume they need all 10 chips for their 8B q3 model. Otherwise, they would have said so or they would have put a more impressive model as the demo. https://www.nextplatform.com/2026/02/19/taalas-etches-ai-mod...

It doesn’t make any sense to think you need the whole server to run one model. It’s much more likely that each server runs 10 instances of the model 1. It doesn’t make sense in terms of architecture. It’s one chip. You can’t split one model over 10 identical hardwire chips 2. It doesn’t add up with their claims of better power efficiency. 2.4kW for one model would be really bad.

Thanks for having a brain.

Not sure who started that "split into 10 chips" claim, it's just dumb.

This is Llama 3B hardcoded (literally) on one chip. That's what the startup is about, they emphasize this multiple times.

Re: The path to ubiquitous AI (17k tokens/sec)

#96

Post by freeone3000 »

danpalmer wrote: post #28
aurareturn wrote: post #8

Edit: it seems like this is likely one chip and not 10. I assumed 8B 16bit quant with 4K or more context. This made me think that they must have chained multiple chips together since N6 850mm2 chip would only yield 3GB of SRAM max. Instead, they seem to have etched llama 8B q3 with 1k context instead which would indeed fit the chip size. This requires 10 chips for an 8 billion q3 param model. 2.4kW. 10 reticle sized…

Alternatively, you could run far more RAG and thinking to integrate recent knowledge, I would imagine models designed for this putting less emphasis on world knowledge and more on agentic search.

Maybe; models with more embedded associations are also better at search. (Intuitively, this tracks; a model with no world knowledge has no awareness of synonyms or relations (a pure markov model), so the more knowledge a model has, the better it can search.) It’s not clear if it’s possible to build such a model, since there doesn’t seem to be a scaling cliff.

Re: The path to ubiquitous AI (17k tokens/sec)

#97

Post by IshKebab »

djb_hackernews wrote: post #70
kleiba wrote: post #48

Earlier quoted context omitted.

Yes, but the quality of the output leaves to be desired. I just asked about some sports history and got a mix of correct information and totally made up nonsense. Not unexpected for an 8k model, but raises the question of what the use case is for such small models.

You have a misunderstanding of what LLMs are good at.

I don't think he does. Larger models are definitely better at not hallucinating. Enough that they are good at answering questions on popular topics.

Smaller models, not so much.

Re: The path to ubiquitous AI (17k tokens/sec)

#98

Post by freeone3000 »

trentnix wrote: post #89

The speed of the chatbot's response is startling when you're used to the simulated fast typing of ChatGPT and others. But the Llama 3.1 8B model Taalas uses predictably results in incorrect answers, hallucinations, poor reliability as a chatbot. What type of latency-sensitive applications are appropriate for a small-model, high-throughput solution like this? I presume this type of specialization is necessary for robo…

Maybe summarization? I’d still worry about accuracy but smaller models do quite well.

Re: The path to ubiquitous AI (17k tokens/sec)

#100

Post by shevy-java »

"Many believe AI is the real deal. In narrow domains, it already surpasses human performance. Used well, it is an unprecedented amplifier of human ingenuity and productivity."

Sounds like people drinking the Kool-Aid now.

I don't reject that AI has use cases. But I do reject that it is promoted as "unprecedented amplifier" of human xyz anything. These folks would even claim how AI improves human creativity. Well, has this been the case?

Post reply on HN