Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The path to ubiquitous AI (17k tokens/sec)

taalas.com

Post reply on HN
Print view View original

271–280 of 471 posts

Re: The path to ubiquitous AI (17k tokens/sec)

#271

Post by condiment »

Gareth321 wrote: post #158
vessenes wrote: post #85

Earlier quoted context omitted.

This math is useful. Lots of folks scoffing in the comments below. I have a couple reactions, after chatting with it: 1) 16k tokens / second is really stunningly fast. There’s an old saying about any factor of 10 being a new science / new product category, etc. This is a new product category in my mind, or it could be. It would be incredibly useful for voice agent applications, realtime loops, realtime video generati…

I think the next major innovation is going to be intelligent model routing. I've been exploring OpenClaw and OpenRouter, and there is a real lack of options to select the best model for the job and execute. The providers are trying to do that with their own models, but none of them offer everything to everyone at all times. I see a future with increasingly niche models being offered for all kinds of novel use cases.…

At 16k tokens/s why bother routing? We're talking about multiple orders of magnitude faster and cheaper execution.

Abundance supports different strategies. One approach: Set a deadline for a response, send the turn to every AI that could possibly answer, and when the deadline arrives, cancel any request that hasn't yet completed. You know a priori which models have the highest quality in aggregate. Pick that one.

Re: The path to ubiquitous AI (17k tokens/sec)

#272

Post by weli »

tgsovlerkhgsel wrote: post #268

Their "chat jimmy" demo sure is fast, but it's not useful at all. Test prompt: ``` Please classify the sentiment of this post as "positive", "neutral" or "negative": Given the price, I expected very little from this case, and I was 100% right. ``` Jimmy: Neutral. I tried various other examples that I had successfully "solved" with very early LLMs and the results were similarly bad.

Maybe its the tism but I also read that sentence as neutral. You expected very little and you got very little. Why would that be positive or negative? Maybe it should be positive because you got what you were expecting? But I would call getting what you expect something neutral, if you expected little and got a lot then that would be positive. If you expected a lot and got little then its negative. But if you expected little and got little the most clear outcome is that its a neutral statement. Am I missing something?

Re: The path to ubiquitous AI (17k tokens/sec)

#273

Post by ttul »

rbanffy wrote: post #232
vessenes wrote: post #85

Earlier quoted context omitted.

This math is useful. Lots of folks scoffing in the comments below. I have a couple reactions, after chatting with it: 1) 16k tokens / second is really stunningly fast. There’s an old saying about any factor of 10 being a new science / new product category, etc. This is a new product category in my mind, or it could be. It would be incredibly useful for voice agent applications, realtime loops, realtime video generati…

> any factor of 10 being a new science / new product category, I often remind people two orders of quantitative change is a qualitative change. > The thing that I’m really very skeptical of is the 2 month turnaround. To get leading edge geometry turned around on arbitrary 2 month schedules is .. ambitious. Hopeful. We could use other words as well. The real product they have is automation. They figured out a way to c…

Having dabbled in VLSI in the early-2010s, half the battle is getting a manufacturing slot with TSMC. It’s a dark art with secret handshakes. This demonstrator chip is an enormous accomplishment.

Re: The path to ubiquitous AI (17k tokens/sec)

#275

Post by robotpepi »

johnsimer wrote: post #249
audunw wrote: post #160

Earlier quoted context omitted.

Models don’t get old as fast as they used to. A lot of the improvements seem to go into making the models more efficient, or the infrastructure around the models. If newer models mainly compete on efficiency it means you can run older models for longer on more efficient hardware while staying competitive. If power costs are significantly lower, they can pay for themselves by the time they are outdated. It also means…

“ Models don’t get old as fast as they used to” ^^^ I think the opposite is true Anthropic and OpenAI are releasing new versions every 60-90 days it seems now, and you could argue they’re going to start releasing even faster

Are they becoming better at the same rate as before though?

Re: The path to ubiquitous AI (17k tokens/sec)

#276

Post by sfn42 »

adamtaylor_13 wrote: post #218
varispeed wrote: post #65

Earlier quoted context omitted.

There is nothing smart about current LLMs. They just regurgitate text compressed in their memory based on probability. None of the LLMs currently have actual understanding of what you ask them to do and what they respond with.

If LLMs just regurgitate compressed text, they'd fail on any novel problem not in their training data. Yet, they routinely solve them, which means whatever's happening between input and output is more than retrieval, and calling it "not understanding" requires you to define understanding in a way that conveniently excludes everything except biological brains.

Yes there are some fascinating emergent properties at play, but when they fail it's blatantly obvious that there's no actual intelligence nor understanding. They are very cool and very useful tools, I use them on a daily basis now and the way I can just paste a vague screenshot with some vague text and they get it and give a useful response blows my mind every time. But it's very clear that it's all just smoke and mirrors, they're not intelligent and you can't trust them with anything.

Re: The path to ubiquitous AI (17k tokens/sec)

#277

Post by ThePhysicist »

freakynit wrote: post #202
ThePhysicist wrote: post #170

This is really cool! I am trying to find a way to accelerate LLM inference for PII detection purposes, where speed is really necessary as we want to process millions of log lines per minute, I am wondering how fast we could get e.g. llama 3.1 to run on a conventional NVIDIA card? 10k tokens per second would be fantastic but even at 1k this would be very useful.

PII redaction is a really good use-case. Also, "10k tokens per second would be fantastic" might not be sufficient (even remotely) if you want to "process millions of log lines per minute". Assuming a single log line at just 100 tokens, you need (100 * 2 million / 60) ~ 3.3 million tokens per second processing speed :)

Yeah I mean we have a mechanism that can bypass AI models for log lines where we are pretty sure no PII is in there (kind of like smart caching using fuzzy template matching to identify things that we have seen before many times, as logs tend to contain the same stuff over and over with tiny variations e.g. different timestamps), so we only need to pass the lines where we cannot be sure there's nothing to the AI for inspection. And we can of course parallelize. Currently we use a homebrew CFR model with lots of tweaks and it's quite good but an LLM would of course be much better still and capture a lof of cases that would evade the simpler model.

Re: The path to ubiquitous AI (17k tokens/sec)

#278

Post by throwaw12 »

segmondy wrote: post #274

Pretty cool, what they need is to build a tool that can take any model to chip in short a time as possible. How quick can they give me DeepSeek, Kimi, Qwen or GLM on a chip? I'll take 5k tk/sec for those!

also imagine it will cost 300$/unit, we all will host our own set of models locally, dream dream

Re: The path to ubiquitous AI (17k tokens/sec)

#279

Post by [deleted] »

weli wrote: post #272
tgsovlerkhgsel wrote: post #268

Their "chat jimmy" demo sure is fast, but it's not useful at all. Test prompt: ``` Please classify the sentiment of this post as "positive", "neutral" or "negative": Given the price, I expected very little from this case, and I was 100% right. ``` Jimmy: Neutral. I tried various other examples that I had successfully "solved" with very early LLMs and the results were similarly bad.

Maybe its the tism but I also read that sentence as neutral. You expected very little and you got very little. Why would that be positive or negative? Maybe it should be positive because you got what you were expecting? But I would call getting what you expect something neutral, if you expected little and got a lot then that would be positive. If you expected a lot and got little then its negative. But if you expecte…

[deleted]
Post reply on HN