Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Real-time LLM Inference on Standard GPUs: 3k tokens/s per request

blog.kog.ai

Post reply on HN
Print view View original

71–80 of 108 posts

Re: Real-time LLM Inference on Standard GPUs: 3k tokens/s per request

#71

Post by hannune »

gaeld wrote: post #65
frankensteins wrote: post #48

I have a naive question here - first, the token speed is very impressive. but why this is the highlight? I would prefer the actual performance.

Token generation speed matters for sequential agentic workflows, like software engineering / vibe coding, where a lot of reasoning tokens, code generation, refactoring, testing, etc. happen in a loop before an actual outcome is served to the user. About model performance, we plan to support the latest frontier models (this tech preview is about the speed of the engine)

[flagged]

Re: Real-time LLM Inference on Standard GPUs: 3k tokens/s per request

#72

Post by __natty__ »

rashkov wrote: post #58

Don't miss trying their demo: https://playground.kog.ai/ Feels like a preview of the future

As for the demo it's fast and extremely dumb like expected for 2B. I asked how to stop drinking habit and in just one follow-up message it recommended trying 8% ABV. Hilarious.

Re: Real-time LLM Inference on Standard GPUs: 3k tokens/s per request

#73

Post by gaeld »

__natty__ wrote: post #72
rashkov wrote: post #58

Don't miss trying their demo: https://playground.kog.ai/ Feels like a preview of the future

As for the demo it's fast and extremely dumb like expected for 2B. I asked how to stop drinking habit and in just one follow-up message it recommended trying 8% ABV. Hilarious.

it's also a coding model

Re: Real-time LLM Inference on Standard GPUs: 3k tokens/s per request

#74

Post by KronisLV »

rashkov wrote: post #58

Don't miss trying their demo: https://playground.kog.ai/ Feels like a preview of the future

You can get something pretty fast right now with a Cerebras Coder subscription, sadly I think the best model they had last I checked was the somewhat dated GLM 4.7: https://inference-docs.cerebras.ai/models/overview

I feel like if they got DeepSeek V4 Flash and Pro running on their hardware, even if at less than 1000 tok/s, they’d still be crushing it with any subscription they’d provide, given how generous their token limits were.

Re: Real-time LLM Inference on Standard GPUs: 3k tokens/s per request

#75

Post by nethi »

gaeld wrote: post #73
__natty__ wrote: post #72

Earlier quoted context omitted.

As for the demo it's fast and extremely dumb like expected for 2B. I asked how to stop drinking habit and in just one follow-up message it recommended trying 8% ABV. Hilarious.

it's also a coding model

Nah. it says it can't even write python code

Re: Real-time LLM Inference on Standard GPUs: 3k tokens/s per request

#78

Post by averne_ »

nightski wrote: post #67
gaeld wrote: post #21

Earlier quoted context omitted.

I guessed you thought about consumer GPUs. We are about standard datacenter GPUs indeed.

How would you classify a datacenter GPU as standard/non-standard? That doesn't seem to be a meaningful distinction. It's click bait.

The blog makes it clear that "standard" GPU here is in opposition to purpose-built hardware like Cerebras. The selling point is reaching the same order of magnitude in generative speed as those approaches.

Re: Real-time LLM Inference on Standard GPUs: 3k tokens/s per request

#79

Post by joefourier »

Do you think the work will still apply to speculative/alternative decoding methods like MTP and block diffusion, which are making batch=1 decoding less memory bound? Kernel launch overhead and memory transfer become less and less significant as a % of time when computing multiple tokens at once.

Re: Real-time LLM Inference on Standard GPUs: 3k tokens/s per request

#80

Post by dr_kiszonka »

gaeld wrote: post #32
ismailmaj wrote: post #26

Earlier quoted context omitted.

I expected a 4090, maybe 2. I did not expect 8xH200 for a 2B model.

Great points, let me clarify: - model size: 2B is just for this preview (it was faster to implement), our article explains how we expect to support large frontier MoE at 1,000 to 5,000 tokens/s - reaching 500 tok/s, or even up to ~1,000 tok/s, on a consumer GPU card is possible with existing inference engines like vLLM. But there is a ceiling. The hard part comes we you try to be faster than that: these frameworks wo…

Thank you for explaining. Do you think there are still opportunities for stack optimizations to meaningfully speed up inference on single consumer-grade GPUs?
Post reply on HN