Running GPT-OSS-120B at 500 tokens per second on Nvidia GPUs
1–10 of 180 posts
Re: Running GPT-OSS-120B at 500 tokens per second on Nvidia GPUs
#2[deleted]
Re: Running GPT-OSS-120B at 500 tokens per second on Nvidia GPUs
#3Such a fascinating read. I didn't realize how much massaging needed to be done to get the models to perform well. I just sort of assumed they worked out of the box.
Re: Running GPT-OSS-120B at 500 tokens per second on Nvidia GPUs
#4[deleted]
Re: Running GPT-OSS-120B at 500 tokens per second on Nvidia GPUs
#5Maybe I'm especially daft this morning but I don't get the point of the speculative decoding.
How does the target model validate the draft tokens without running the inference as normal?
Because if it is doing just that, I don't get the point as you can't trust the draft tokens before they are validated, so you're still stuck waiting for the target model.
Re: Running GPT-OSS-120B at 500 tokens per second on Nvidia GPUs
#6Maybe I'm especially daft this morning but I don't get the point of the speculative decoding. How does the target model validate the draft tokens without running the inference as normal? Because if it is doing just that, I don't get the point as you can't trust the draft tokens before they are validated, so you're still stuck waiting for the target model.
It does run inference, but on the batch of tokens that were drafted, akin to the prefill phase.
So your draft model can decode N new tokens, then the real model does one inference pass to score the N new drafted tokens.
Prefill is computation bound whereas decode is bandwidth bound, so in practice doing one prefill over N tokens is cheaper than doing N decode passes.
Re: Running GPT-OSS-120B at 500 tokens per second on Nvidia GPUs
#7Maybe I'm especially daft this morning but I don't get the point of the speculative decoding. How does the target model validate the draft tokens without running the inference as normal? Because if it is doing just that, I don't get the point as you can't trust the draft tokens before they are validated, so you're still stuck waiting for the target model.
Not an expert, but here's how I understand it. You know how input tokens are cheaper than output tokens? It's related to that.
Say the model so far has "The capital of France". The small model generates "is Paris.", which let's say is 5 tokens.
You feed the large model "The capital of France is Paris." to validate all 5 of those tokens in a single forward pass.
Re: Running GPT-OSS-120B at 500 tokens per second on Nvidia GPUs
#8What's the best speed people have gotten on 4090s?
Re: Running GPT-OSS-120B at 500 tokens per second on Nvidia GPUs
#9Maybe I'm especially daft this morning but I don't get the point of the speculative decoding. How does the target model validate the draft tokens without running the inference as normal? Because if it is doing just that, I don't get the point as you can't trust the draft tokens before they are validated, so you're still stuck waiting for the target model.
Not an expert, but here's how I understand it. You know how input tokens are cheaper than output tokens? It's related to that. Say the model so far has "The capital of France". The small model generates "is Paris.", which let's say is 5 tokens. You feed the large model "The capital of France is Paris." to validate all 5 of those tokens in a single forward pass.
But what would happen if the small model's prediction was "is Rome."? Wouldn't that result in costlier inference if the small model is "wrong" more than it is correct.
Also, if the small model would be sufficiently more "correct" than "wrong", wouldn't be more efficient to get rid of the large model at this point?
Re: Running GPT-OSS-120B at 500 tokens per second on Nvidia GPUs
#10Maybe I'm especially daft this morning but I don't get the point of the speculative decoding. How does the target model validate the draft tokens without running the inference as normal? Because if it is doing just that, I don't get the point as you can't trust the draft tokens before they are validated, so you're still stuck waiting for the target model.
My simplified understanding: The target model can validate the draft tokens all at once, in a single forward pass. The output of that forward pass is a list of probabilities for each draft token which are compared to the probabilities produced by the draft model. If the target model's probabilities are the same or greater than the draft model, the tokens are accepted. Worst case none of the draft tokens are accepted and instead the target model selects the single next token as usual.