Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The path to ubiquitous AI (17k tokens/sec)

taalas.com

Post reply on HN
Print view View original

341–350 of 471 posts

Re: The path to ubiquitous AI (17k tokens/sec)

#341

Post by ACCount37 »

MITSardine wrote: post #135
ACCount37 wrote: post #105

Earlier quoted context omitted.

What uses, exactly? The prototype is: silicon with a Llama 3.1 8B etched into it. Today's 4B models already outperform it. Token rate in five digits is a major technical flex, but, does anyone really need to run a very dumb model at this speed? The only things that come to mind that could reap a benefit are: asymmetric exotics like VLA action policies and voice stages for V2V models. Both of which are "small fast low…

With LLMs this fast, you could imagine using them as any old function in programs.

You could always have. Assuming you have an API or a local model.

Which was always the killer assumption, and this changes little.

Re: The path to ubiquitous AI (17k tokens/sec)

#343

Post by dave1010uk »

Retr0id wrote: post #239
smusamashah wrote: post #221

Earlier quoted context omitted.

With this speed, you can keep looping and generating code until it passes all tests. If you have tests. Generate lots of solutions and mix and match. This allows a new way to look at LLMs.

Not just looping, you could do a parallel graph search of the solution-space until you hit one that works.

You could also parse prompts into an AST, run inference, run evals, then optimise the prompts with something like a genetic algorithm.

Re: The path to ubiquitous AI (17k tokens/sec)

#344

Post by AlexC04 »

If I could have one of these cards in my own computer do you think it would be possible to replace claude code?

1. Assume It's running a better model, even a dedicated coding model. High scoring but obviously not opus 4.5 2. Instead of the standard send-receive paradigm we set up a pipeline of agents, each of whom parses the output of the previous.

At 17k/tps running locally, you could effectively spin up tasks like "you are an agent who adds semicolons to the end of the line in javascript", with some sort of dedicated software in the style of claude code you could load an array of 20 agents each with a role to play in improving outpus.

take user input and gather context from codebase -> rewrite what you think the human asked you in the form of an LLM-optimized instructional prompt -> examine the prompt for uncertainties and gaps in your understanding or ability to execute -> -> execute the work

Could you effectively set up something that is configurable to the individual developer - a folder of system prompts that every request loops through?

Do you really need the best model if you can pass your responses through a medium tier model that engages in rapid self improvement 30 times in a row before your claude server has returned its first shot response?

Re: The path to ubiquitous AI (17k tokens/sec)

#346

Post by AmazingTurtle »

AlexC04 wrote: post #344

If I could have one of these cards in my own computer do you think it would be possible to replace claude code? 1. Assume It's running a better model, even a dedicated coding model. High scoring but obviously not opus 4.5 2. Instead of the standard send-receive paradigm we set up a pipeline of agents, each of whom parses the output of the previous. At 17k/tps running locally, you could effectively spin up tasks like…

Models can't improve themselves with their own (model) input, they need to be grounded in truth and reality.

Re: The path to ubiquitous AI (17k tokens/sec)

#347

Post by turnsout »

robotpepi wrote: post #275
johnsimer wrote: post #249

Earlier quoted context omitted.

“ Models don’t get old as fast as they used to” ^^^ I think the opposite is true Anthropic and OpenAI are releasing new versions every 60-90 days it seems now, and you could argue they’re going to start releasing even faster

Are they becoming better at the same rate as before though?

yes, pretty much

Re: The path to ubiquitous AI (17k tokens/sec)

#348

Post by ethmarks »

jtr1 wrote: post #328
jjcm wrote: post #112

A lot of naysayers in the comments, but there are so many uses for non-frontier models. The proof of this is in the openrouter activity graph for llama 3.1: https://openrouter.ai/meta-llama/llama-3.1-8b-instruct/activ... 10b daily tokens growing at an average of 22% every week . There are plenty of times I look to groq for narrow domain responses - these smaller models are fantastic for that and there's often no need…

Maybe this is a naive question, but why wouldn't there be market for this even for frontier models? If Anthropic wanted to burn Opus 4.6 into a chip, wouldn't there theoretically be a price point where this would lower inference costs for them?

Because we don't know if this would scale well to high-quality frontier models. If you need to manufacture dedicated hardware for each new model, that adds a lot of expense and causes a lot of e-waste once the next model releases. In contrast, even this current iteration seems like it would be fantastic for low-grade LLM work.

For example, searching a database of tens of millions of text files. Very little "intelligence" is required, but cost and speed are very important. If you want to know something specific on Wikipedia but don't want to figure out which article to search for, you can just have an LLM read the entire English Wikipedia (7,140,211 articles) and compile a report. Doing that would be prohibitively expensive and glacially slow with standard LLM providers, but Taalas could probably do it in a few minutes or even seconds, and it would probably be pretty cheap.

Re: The path to ubiquitous AI (17k tokens/sec)

#349

Post by heliumtera »

Yep, this is the most exciting demo for me yet. Holy cow this is unbelievably fast.

The most impressive demo since gpt 3, honestly.

Since we already have open source models that are plenty good, like the new kimi k2.5, all I need is the ability to run it at moderate speed.

Honestly I am not bullish on capabilities that models do not yet have, seems we have seen it all and the only advancement have been context size.

And honestly I would claim this is the market sentiment aswell, anthropic showed opus 4.6 first and the big release was actually sonnet, the model people would use routinely. Nobody gave a shit about Gemini 3.1 pro, 3.0 flash was very successful...

Given all the recent developments in the last 12 months, no new use cases have opened for me. Given this insane speed, even on a limited model/context size, we would approach IA very differently.

Re: The path to ubiquitous AI (17k tokens/sec)

#350

Post by wmf »

freakynit wrote: post #136
amelius wrote: post #128

Earlier quoted context omitted.

OK investors, time to pull out of OpenAI and move all your money to ChatJimmy.

A related argument I raised a few days back on HN: What's the moat with with these giant data-centers that are being built with 100's of billions of dollars on nvidia chips? If such chips can be built so easily, and offer this insane level of performance at 10x efficiency, then one thing is 100% sure: more such startups are coming... and with that, an entire new ecosystem.

Nvidia bought all the capacity so their competitors can't be manufactured at scale.
Post reply on HN