Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Running GPT-OSS-120B at 500 tokens per second on Nvidia GPUs

baseten.co

Post reply on HN
Print view View original

11–20 of 180 posts

Re: Running GPT-OSS-120B at 500 tokens per second on Nvidia GPUs

#11

Post by isoprophlex »

furyofantares wrote: post #7
magicalhippo wrote: post #5

Maybe I'm especially daft this morning but I don't get the point of the speculative decoding. How does the target model validate the draft tokens without running the inference as normal? Because if it is doing just that, I don't get the point as you can't trust the draft tokens before they are validated, so you're still stuck waiting for the target model.

Not an expert, but here's how I understand it. You know how input tokens are cheaper than output tokens? It's related to that. Say the model so far has "The capital of France". The small model generates "is Paris.", which let's say is 5 tokens. You feed the large model "The capital of France is Paris." to validate all 5 of those tokens in a single forward pass.

but... do you get any validation during the forward pass? the small model could just as well have generated "is Berlin." or whatever. do these models somehow give you a likelihood for the next token when you're prefilling, that you can compare against? if so why not just... use that always?

or is this a scenario where computation is expensive but validation is cheap?

EDIT: thanks, people, for educating me! very insightful :)

Re: Running GPT-OSS-120B at 500 tokens per second on Nvidia GPUs

#13

Post by robrenaud »

magicalhippo wrote: post #5

Maybe I'm especially daft this morning but I don't get the point of the speculative decoding. How does the target model validate the draft tokens without running the inference as normal? Because if it is doing just that, I don't get the point as you can't trust the draft tokens before they are validated, so you're still stuck waiting for the target model.

I think your core misunderstanding is that you are assuming K calls to generate 1 token is expensive as 1 call to generate K tokens. It is actually much more expensive to generate serially than even in small batches.

Re: Running GPT-OSS-120B at 500 tokens per second on Nvidia GPUs

#15

Post by cwyers »

ahmedfromtunis wrote: post #9
furyofantares wrote: post #7

Earlier quoted context omitted.

Not an expert, but here's how I understand it. You know how input tokens are cheaper than output tokens? It's related to that. Say the model so far has "The capital of France". The small model generates "is Paris.", which let's say is 5 tokens. You feed the large model "The capital of France is Paris." to validate all 5 of those tokens in a single forward pass.

But what would happen if the small model's prediction was "is Rome."? Wouldn't that result in costlier inference if the small model is "wrong" more than it is correct. Also, if the small model would be sufficiently more "correct" than "wrong", wouldn't be more efficient to get rid of the large model at this point?

So, the way speculative decoding works, the model begins predicting at the first wrong token, so you still get 'is' for free.

Re: Running GPT-OSS-120B at 500 tokens per second on Nvidia GPUs

#16

Post by porridgeraisin »

magicalhippo wrote: post #5

Maybe I'm especially daft this morning but I don't get the point of the speculative decoding. How does the target model validate the draft tokens without running the inference as normal? Because if it is doing just that, I don't get the point as you can't trust the draft tokens before they are validated, so you're still stuck waiting for the target model.

Let's say I want to run f2(f1(x)) where f1 and f2 are both a single pass through GPT4.

This takes 2 seconds time, assuming 1 second for every pass.

What I instead do is kick off f1(x) in another thread, and then run f2(g1(x)) where g1 is one pass through GPT-nano.

This takes 1 + 0.1 seconds, assuming gpt nano takes 0.1s for every pass. In this 1.1 seconds, the f1(x) that we kicked off in the 2nd thread would have finished (it takes 1 second).

So in 1.1 seconds we have available to us f1(x), f2(g1(x)), and we store the intermediate g1(x) as well

We compare g1(x) and f1(x)

If they were equal, i.e g1(x) = f1(x), then we have our answer = f2(g1(x)) in just 1.1s.

If they were not, we compute f2(output of f1(x) from 2nd thread) which takes 1 further second, bringing our total to 2.1s.

If the small model is equalling the big model in say 2/3 of cases, you will spend 2/3 * 1.1 + 1/3 * 2.1 = 1.433s on average for this computation. Without speculative decoding, it is always 2s.

Re: Running GPT-OSS-120B at 500 tokens per second on Nvidia GPUs

#18

Post by acters »

tmshapland wrote: post #3

Such a fascinating read. I didn't realize how much massaging needed to be done to get the models to perform well. I just sort of assumed they worked out of the box.

Personally, I think bigger companies should be more proactive and work with some of the popular inference engine software devs with getting their special snowflake LLM to work before it gets released. I guess it is all very much experimental at the end of the day. Those devs are putting in God's work for us to use on our budget friendly hardware choices.

Re: Running GPT-OSS-120B at 500 tokens per second on Nvidia GPUs

#19

Post by arkmm »

porridgeraisin wrote: post #16
magicalhippo wrote: post #5

Maybe I'm especially daft this morning but I don't get the point of the speculative decoding. How does the target model validate the draft tokens without running the inference as normal? Because if it is doing just that, I don't get the point as you can't trust the draft tokens before they are validated, so you're still stuck waiting for the target model.

Let's say I want to run f2(f1(x)) where f1 and f2 are both a single pass through GPT4. This takes 2 seconds time, assuming 1 second for every pass. What I instead do is kick off f1(x) in another thread, and then run f2(g1(x)) where g1 is one pass through GPT-nano. This takes 1 + 0.1 seconds, assuming gpt nano takes 0.1s for every pass. In this 1.1 seconds, the f1(x) that we kicked off in the 2nd thread would have fin…

This is a really great explanation.

Re: Running GPT-OSS-120B at 500 tokens per second on Nvidia GPUs

#20

Post by [deleted] »

ahmedfromtunis wrote: post #9
furyofantares wrote: post #7

Earlier quoted context omitted.

Not an expert, but here's how I understand it. You know how input tokens are cheaper than output tokens? It's related to that. Say the model so far has "The capital of France". The small model generates "is Paris.", which let's say is 5 tokens. You feed the large model "The capital of France is Paris." to validate all 5 of those tokens in a single forward pass.

But what would happen if the small model's prediction was "is Rome."? Wouldn't that result in costlier inference if the small model is "wrong" more than it is correct. Also, if the small model would be sufficiently more "correct" than "wrong", wouldn't be more efficient to get rid of the large model at this point?

[deleted]
Post reply on HN