Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The path to ubiquitous AI (17k tokens/sec)

taalas.com

Post reply on HN
Print view View original

111–120 of 471 posts

Re: The path to ubiquitous AI (17k tokens/sec)

#111

Post by dust42 »

zozbot234 wrote: post #104
dust42 wrote: post #90

Earlier quoted context omitted.

Low latency inference is very useful in voice-to-voice applications. You say it is a waste of power but at least their claim is that it is 10x more efficient. We'll see but if it works out it will definitely find its applications.

This is not voice-to-voice though, end-to-end voice chat models (the Her UX) are completely different.

I haven't found any end-to-end voice chat models useful. I had much better results with separate STT-LLM-TTS. One big problem is the turn detection and having inference with 150-200ms latency would allow for a whole new level of quality. I would just use it with a prompt: "You think the user is finished talking?" and then push it to a larger model. The AI should reply within the ballpark of 600ms-1000ms. Faster is often irritating, slower will make the user to start talking again.

Re: The path to ubiquitous AI (17k tokens/sec)

#112

Post by jjcm »

A lot of naysayers in the comments, but there are so many uses for non-frontier models. The proof of this is in the openrouter activity graph for llama 3.1: https://openrouter.ai/meta-llama/llama-3.1-8b-instruct/activ...

10b daily tokens growing at an average of 22% every week.

There are plenty of times I look to groq for narrow domain responses - these smaller models are fantastic for that and there's often no need for something heavier. Getting the latency of reponses down means you can use LLM-assisted processing in a standard webpage load, not just for async processes. I'm really impressed by this, especially if this is its first showing.

Re: The path to ubiquitous AI (17k tokens/sec)

#114

Post by 100721 »

small_model wrote: post #84
varispeed wrote: post #65

Earlier quoted context omitted.

There is nothing smart about current LLMs. They just regurgitate text compressed in their memory based on probability. None of the LLMs currently have actual understanding of what you ask them to do and what they respond with.

Thats not how they work, pro-tip maybe don't comment until you have a good understanding?

Huh? Their words are an accurate, if simplified, description of how they work.

Re: The path to ubiquitous AI (17k tokens/sec)

#115

Post by freakynit »

trentnix wrote: post #89

The speed of the chatbot's response is startling when you're used to the simulated fast typing of ChatGPT and others. But the Llama 3.1 8B model Taalas uses predictably results in incorrect answers, hallucinations, poor reliability as a chatbot. What type of latency-sensitive applications are appropriate for a small-model, high-throughput solution like this? I presume this type of specialization is necessary for robo…

You could build realtime API routing and orchestration systems that rely on high quality language understanding but need near-instant responses. Examples:

1. Intent based API gateways: convert natural language queries into structured API calls in real time (eg., "cancel my last order and refund it to the original payment method" -> authentication, order lookup, cancellation, refund API chain).

2. Of course, realtime voice chat.. kinda like you see in movies.

3. Security and fraud triage systems: parse logs without hardcoded regexes and issue alerts and full user reports in real time and decide which automated workflows to trigger.

4. Highly interactive what-if scenarios powered by natural language queries.

This effectively gives you database level speeds on top of natural language understanding.

Re: The path to ubiquitous AI (17k tokens/sec)

#116

Post by 9cb14c1ec0 »

aurareturn wrote: post #18
grzracz wrote: post #13

This would be killer for exploring simultaneous thinking paths and council-style decision taking. Even with Qwen3-Coder-Next 80B if you could achieve a 10x speed, I'd buy one of those today. Can't wait to see if this is still possible with larger models than 8B.

It uses 10 chips for 8B model. It’d need 80 chips for an 80b model. Each chip is the size of an H100. So 80 H100 to run at this speed. Can’t change the model after you manufacture the chips since it’s etched into silicon.

As many others in this conversation have asked, can we have some sources on the idea that the model is spread across chips? You keep making the claim, but no one (myself included) else has any idea where that information comes from or if it is correct.

Re: The path to ubiquitous AI (17k tokens/sec)

#117

Post by small_model »

faeyanpiraat wrote: post #102
shevy-java wrote: post #100

"Many believe AI is the real deal. In narrow domains, it already surpasses human performance. Used well, it is an unprecedented amplifier of human ingenuity and productivity." Sounds like people drinking the Kool-Aid now. I don't reject that AI has use cases. But I do reject that it is promoted as "unprecedented amplifier" of human xyz anything. These folks would even claim how AI improves human creativity. Well, has…

For me, this is entirely true. I'm progressing with my side projects like I've never before.

Same, I would have given up on them long ago, I no longer code at all now. Why would I when the latest models can do it better, faster and without the human limitations of tiredness, emotional impacts etc.

Re: The path to ubiquitous AI (17k tokens/sec)

#119

Post by VMG »

grzracz wrote: post #24
metabrew wrote: post #5

I tried the chatbot. jarring to see a large response come back instantly at over 15k tok/sec I'll take one with a frontier model please, for my local coding and home ai needs..

Absolute insanity to see a coherent text block that takes at least 2 minutes to read generated in a fraction of a second. Crazy stuff...

Not at all if you consider the internet pre-LLM. That is the standard expectation when you load a website.

The slow word-by-word typing was what we started to get used to with LLMs.

If these techniques get widespread, we may grow accustomed to the "old" speed again where content loads ~instantly.

Imagine a content forest like Wikipedia instantly generated like a Minecraft word...

Post reply on HN