Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Running GPT-OSS-120B at 500 tokens per second on Nvidia GPUs

baseten.co

Post reply on HN
Print view View original

21–30 of 180 posts

Re: Running GPT-OSS-120B at 500 tokens per second on Nvidia GPUs

#21

Post by acters »

ahmedfromtunis wrote: post #9
furyofantares wrote: post #7

Earlier quoted context omitted.

Not an expert, but here's how I understand it. You know how input tokens are cheaper than output tokens? It's related to that. Say the model so far has "The capital of France". The small model generates "is Paris.", which let's say is 5 tokens. You feed the large model "The capital of France is Paris." to validate all 5 of those tokens in a single forward pass.

But what would happen if the small model's prediction was "is Rome."? Wouldn't that result in costlier inference if the small model is "wrong" more than it is correct. Also, if the small model would be sufficiently more "correct" than "wrong", wouldn't be more efficient to get rid of the large model at this point?

I believe that is exactly the downside of using speculative decoding, which is why it is very important to have the models properly sized between each other by making sure the small use is big enough to be mostly correct while also being exceptionally faster than the larger one. However the larger one has to be fast enough that catching flaws won't introduce too manyrandom delays. Also, if the small one is incorrect then the larger one correcting the mistake is miles better than leaving in incorrect output.

It is about improving quality while allowing for faster speed most of the time. The tradeoff is that you consume more memory from having two models loaded vs one of them exclusively.

If you just focus on one then it would make sense to reduce memory usage by just running the smaller model.

Re: Running GPT-OSS-120B at 500 tokens per second on Nvidia GPUs

#22

Post by modeless »

ActorNightly wrote: post #12
modeless wrote: post #8

What's the best speed people have gotten on 4090s?

You can't fit the model into 4090 without quantization, its like 64 gigs. For home use, Gemma27B QAT is king. Its almost as good as Deepseek R1

The 20B one fits.

Re: Running GPT-OSS-120B at 500 tokens per second on Nvidia GPUs

#24

Post by magicalhippo »

porridgeraisin wrote: post #16
magicalhippo wrote: post #5

Maybe I'm especially daft this morning but I don't get the point of the speculative decoding. How does the target model validate the draft tokens without running the inference as normal? Because if it is doing just that, I don't get the point as you can't trust the draft tokens before they are validated, so you're still stuck waiting for the target model.

Let's say I want to run f2(f1(x)) where f1 and f2 are both a single pass through GPT4. This takes 2 seconds time, assuming 1 second for every pass. What I instead do is kick off f1(x) in another thread, and then run f2(g1(x)) where g1 is one pass through GPT-nano. This takes 1 + 0.1 seconds, assuming gpt nano takes 0.1s for every pass. In this 1.1 seconds, the f1(x) that we kicked off in the 2nd thread would have fin…

Thanks, very nice explanation, that makes perfect sense. I guess their graphics confused me for some reason and had me thinking all wrong.

Now I see they tried to point out the obvious thing which is to predict multiple tokens ahead, not just two as in your example.

Re: Running GPT-OSS-120B at 500 tokens per second on Nvidia GPUs

#25

Post by acters »

acters wrote: post #21
ahmedfromtunis wrote: post #9

Earlier quoted context omitted.

But what would happen if the small model's prediction was "is Rome."? Wouldn't that result in costlier inference if the small model is "wrong" more than it is correct. Also, if the small model would be sufficiently more "correct" than "wrong", wouldn't be more efficient to get rid of the large model at this point?

I believe that is exactly the downside of using speculative decoding, which is why it is very important to have the models properly sized between each other by making sure the small use is big enough to be mostly correct while also being exceptionally faster than the larger one. However the larger one has to be fast enough that catching flaws won't introduce too manyrandom delays. Also, if the small one is incorrect…

Another caveat with this method is that both larger and smaller models need to behave very similar because a lot of the savings come from generating the necessary fluff around each detail such as grammar, formatting and words/letters that transition between each other.

Unsurprisingly gpt-oss has both larger and smaller models that work very similarly! Both model sizes are so similar that even if getting a few wrong would not be slowing down the performance enough to equal the speed of the larger model(which is the worst case with this setup). We want the speed of the smaller model as much as possible. That is all

Re: Running GPT-OSS-120B at 500 tokens per second on Nvidia GPUs

#26

Post by bhaney »

magicalhippo wrote: post #5

Maybe I'm especially daft this morning but I don't get the point of the speculative decoding. How does the target model validate the draft tokens without running the inference as normal? Because if it is doing just that, I don't get the point as you can't trust the draft tokens before they are validated, so you're still stuck waiting for the target model.

> How does the target model validate the draft tokens without running the inference as normal?

It does run the inference as normal, just in parallel with the other inferences

> if it is doing just that, I don't get the point

Running inferences in parallel allows you to only read the model weights out of memory only once for N parallel inferences, as opposed to reading them out of memory N times for N serial inferences. Inference is massively bottlenecked by memory bandwidth to the tune of one or two orders of magnitude compared to compute, so this helps a lot.

Re: Running GPT-OSS-120B at 500 tokens per second on Nvidia GPUs

#27

Post by steinvakt2 »

asabla wrote: post #14
modeless wrote: post #8

What's the best speed people have gotten on 4090s?

I'm on a 5090 so it's not apples to apples comparison. But I'm getting ~150t/s for the 20B version using ~16000 context size.

And flash attention doesn't work on 5090 yet, right? So currently 4090 is probably faster, or?

Re: Running GPT-OSS-120B at 500 tokens per second on Nvidia GPUs

#28

Post by littlestymaar »

bhaney wrote: post #26
magicalhippo wrote: post #5

Maybe I'm especially daft this morning but I don't get the point of the speculative decoding. How does the target model validate the draft tokens without running the inference as normal? Because if it is doing just that, I don't get the point as you can't trust the draft tokens before they are validated, so you're still stuck waiting for the target model.

> How does the target model validate the draft tokens without running the inference as normal? It does run the inference as normal, just in parallel with the other inferences > if it is doing just that, I don't get the point Running inferences in parallel allows you to only read the model weights out of memory only once for N parallel inferences, as opposed to reading them out of memory N times for N serial inference…

> Inference is massively bottlenecked by memory bandwidth to the tune of one or two orders of magnitude compared to compute, so this helps a lot.

Nitpick: it's only bottlenecked by memory bandwidth if the batch size is too low (that is: if you don't have many users calling the same model in parallel).

Speculative decoding is just a way of running a single query as if it was parallel queries.

Re: Running GPT-OSS-120B at 500 tokens per second on Nvidia GPUs

#29

Post by steinvakt2 »

modeless wrote: post #22
ActorNightly wrote: post #12

Earlier quoted context omitted.

You can't fit the model into 4090 without quantization, its like 64 gigs. For home use, Gemma27B QAT is king. Its almost as good as Deepseek R1

The 20B one fits.

Does it fit on a 5080 (16gb)?
Post reply on HN