Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Real-time LLM Inference on Standard GPUs: 3k tokens/s per request

blog.kog.ai

Post reply on HN
Print view View original

101–108 of 108 posts

Re: Real-time LLM Inference on Standard GPUs: 3k tokens/s per request

#101

Post by selicos »

gaeld wrote: post #66
deflator wrote: post #51

Earlier quoted context omitted.

What a lot of use on here are salivating for is the ability to run these on prosumer hardware at home. So we tend to jump to the conclusion that "standard" means "consumer-grade" because that's what we want to see. Still, very cool work!

thank you deflator, I understand this now! much appreciated

A consumer "Standard GPU" could mean about a 6-8gb VRAM GPU still in support by the manufacturer, independent of CUDA/etc proprietary technology.

Recent Steam hardware survey top GPU list is:

- RTX 3060 (6 or 12gb VRAM)

- RTX 4060 (8 or 16gb)

- RTX 3050 (6 or 8gb)

- RTX 5070 (12gb)

- RTX 5060 (8gb)

- GTX 1650 (4gb!)

That list only covers about 22% of survey respondents but sets a 6-8gb VRAM baseline for consumer GPUs.

Can this run on an RX 570 8gb form 2017? Maybe that's a ways back. A 1660 6gb from 2019? Intel? They had a decent budget run in recent years.

https://store.steampowered.com/hwsurvey/videocard/

Re: Real-time LLM Inference on Standard GPUs: 3k tokens/s per request

#102

Post by blindr »

gaeld wrote: post #85
blindr wrote: post #83

For me it's 3.4k tok/s of pure nonsense, the model is bad, you tell it it's wrong, it acknowledge it's wrong and repeats the same nonsense. It reminds me my nephew though. Ask it something like: "I want to play the guitar on the surface of the Moon. What speakers do you suggest." and then "But Moon has no atmosphere, how the sound will travel?".

Note that this coding model is trained on programming use cases, and is also not tuned for multi-turn chat. You can ask it to implement an algorithm; we provide suggested prompts you can test. Also, this tech preview is really about the speed of the inference engine (not the model itself) so I'm glad you got 3.4k tok/s!

That's what I tested first and the model failed, even after suggestions that it was wrong and how to fix it's errors.

Re: Real-time LLM Inference on Standard GPUs: 3k tokens/s per request

#103

Post by Terretta »

mungoman2 wrote: post #5

This looks very interesting. Possible to get those rates without exotic hardware. But I have to say that the comparison is not really fair. Comparison is done with a 2 B model vs frontier models that are likely 100s of times larger. Also taalas with their 15000 tok/s inference are suspiciously missing from the comparison. We need to see the comparison with this framework and useful models, which at present seems to m…

> single-request decode speed is now the metric that matters

Benched at 96 input tokens, 4000 output tokens.

Re: Real-time LLM Inference on Standard GPUs: 3k tokens/s per request

#104

Post by dandanua »

irishcoffee wrote: post #20

NVIDIA H200 Is not a standard GPU. 8 of them in a box with a cpu and ram costs close to the same as a house. I am 100% all about using local models instead of sending someone else all my data and paying for the privilege of doing so, this article is misleading. I can get a 27b model to kick out 40 tok/s on 16 gb vram. This is the area ripe for development. If you can’t connect a monitor, it isn’t a standard GPU, at l…

This blog post clearly targets VCs, but what they are doing is legit and can improve the performance of local models on low-end hardware as well, especially since their priority is to optimize non-batched inference.

Re: Real-time LLM Inference on Standard GPUs: 3k tokens/s per request

#105

Post by mrkeen »

> Test the speed in our live coding playground: playground.kog.ai

> Dsatur in Haskell

  #include 
  #include 
  #include 
  using namespace std;
  int main() {
    int n;
    cin >> n;
    vector v(n);
    for (int i = 0; i > v[i];
    sort(v.begin(), v.end());
    int i = 0, j = n - 1;
    while (i
Haha. It was fast though.

Re: Real-time LLM Inference on Standard GPUs: 3k tokens/s per request

#108

Post by dippatel1994 »

kirtivr wrote: post #12
mungoman2 wrote: post #5

This looks very interesting. Possible to get those rates without exotic hardware. But I have to say that the comparison is not really fair. Comparison is done with a 2 B model vs frontier models that are likely 100s of times larger. Also taalas with their 15000 tok/s inference are suspiciously missing from the comparison. We need to see the comparison with this framework and useful models, which at present seems to m…

They got 1K tok/s with Deepseek v4 Pro. That's kinda cool..

Exactly! Any optimization for local inference is a welcome change IMHO!
Post reply on HN