Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Running GPT-OSS-120B at 500 tokens per second on Nvidia GPUs

baseten.co

Post reply on HN
Print view View original

31–40 of 180 posts

Re: Running GPT-OSS-120B at 500 tokens per second on Nvidia GPUs

#31

Post by eric-burel »

acters wrote: post #18
tmshapland wrote: post #3

Such a fascinating read. I didn't realize how much massaging needed to be done to get the models to perform well. I just sort of assumed they worked out of the box.

Personally, I think bigger companies should be more proactive and work with some of the popular inference engine software devs with getting their special snowflake LLM to work before it gets released. I guess it is all very much experimental at the end of the day. Those devs are putting in God's work for us to use on our budget friendly hardware choices.

SMEs are starting to want local LLMs and it's a nightmare to figure what hardware would work for what models. I am asking devs in my hometown to literally visit their installs to figure combos that work.

Re: Running GPT-OSS-120B at 500 tokens per second on Nvidia GPUs

#32

Post by sanxiyn »

isoprophlex wrote: post #11
furyofantares wrote: post #7

Earlier quoted context omitted.

Not an expert, but here's how I understand it. You know how input tokens are cheaper than output tokens? It's related to that. Say the model so far has "The capital of France". The small model generates "is Paris.", which let's say is 5 tokens. You feed the large model "The capital of France is Paris." to validate all 5 of those tokens in a single forward pass.

but... do you get any validation during the forward pass? the small model could just as well have generated "is Berlin." or whatever. do these models somehow give you a likelihood for the next token when you're prefilling, that you can compare against? if so why not just... use that always? or is this a scenario where computation is expensive but validation is cheap? EDIT: thanks, people, for educating me! very insig…

Yes, models give likelihoods you can compare against. No, you can't do that without drafting, because likelihood of token N+2 depends on token N+1. That is, you get P(is, The capital of France) and P(Berlin, The capital of France is), but for the later you need to give "is" as input, you can't do P(Berlin, The Capital of France _).

Re: Running GPT-OSS-120B at 500 tokens per second on Nvidia GPUs

#33

Post by shikon7 »

isoprophlex wrote: post #11
furyofantares wrote: post #7

Earlier quoted context omitted.

Not an expert, but here's how I understand it. You know how input tokens are cheaper than output tokens? It's related to that. Say the model so far has "The capital of France". The small model generates "is Paris.", which let's say is 5 tokens. You feed the large model "The capital of France is Paris." to validate all 5 of those tokens in a single forward pass.

but... do you get any validation during the forward pass? the small model could just as well have generated "is Berlin." or whatever. do these models somehow give you a likelihood for the next token when you're prefilling, that you can compare against? if so why not just... use that always? or is this a scenario where computation is expensive but validation is cheap? EDIT: thanks, people, for educating me! very insig…

Yes, the forward pass does a next token prediction on all input tokens (so we know exactly how many tokens from the small model matched). The expensive thing is not the computation, but the memory bandwidth, as each pass needs to load the model from memory.

If the small model predicts some tokens correctly, you save some passes, at the expense of doing some extra computations when the tokens were not correct.

In any case, each forward pass will give at least one new token.

Re: Running GPT-OSS-120B at 500 tokens per second on Nvidia GPUs

#34

Post by eric-burel »

"Encourage Open-Source and Open-Weight AI" is the part just after "Ensure that Frontier AI Protects Free Speech and American Values" in America's AI Action Plan. I know this is not rational but OpenAI OSS models kinda give me chills as I am reading the Plan in parallel. Anyway I like seeing oss model providers talking about hardware, because that's a limiting point for most developers that are not familiar with this layer.

Re: Running GPT-OSS-120B at 500 tokens per second on Nvidia GPUs

#35

Post by PeterStuer »

steinvakt2 wrote: post #27
asabla wrote: post #14

Earlier quoted context omitted.

I'm on a 5090 so it's not apples to apples comparison. But I'm getting ~150t/s for the 20B version using ~16000 context size.

And flash attention doesn't work on 5090 yet, right? So currently 4090 is probably faster, or?

I don't think the 4090 has native 4bit support, which will probably have a significant impact.

Re: Running GPT-OSS-120B at 500 tokens per second on Nvidia GPUs

#36

Post by DrPhish »

sarthaksoni wrote: post #30

Reading this made me realize how easy it is to set up GPT-OSS 20B in comparison. I had it running on my Mac in five minutes, thanks to Llama.

Its also easy to do 120b on CPU if you have the resources. I had 120b running on my home LLM CPU inference box in just as long as it took to download the GGUFs, git pull and rebuild llama-server. I had it running at 40t/s with zero effort and 50t/s with a brief tweaking. Its just too bad that even the 120b isn't really worth running compared to the other models that are out there.

It really is amazing what ggerganov and the llama.cpp team have done to democratize LLMs for individuals that can't afford a massive GPU farm worth more than the average annual salary.

Re: Running GPT-OSS-120B at 500 tokens per second on Nvidia GPUs

#37

Post by jwitthuhn »

steinvakt2 wrote: post #29
modeless wrote: post #22

Earlier quoted context omitted.

The 20B one fits.

Does it fit on a 5080 (16gb)?

Haven't tried myself but it looks like it probably does. The weight files total 13.8 GB which gives you a little left over to hold your context.

Re: Running GPT-OSS-120B at 500 tokens per second on Nvidia GPUs

#38

Post by jlebar »

magicalhippo wrote: post #5

Maybe I'm especially daft this morning but I don't get the point of the speculative decoding. How does the target model validate the draft tokens without running the inference as normal? Because if it is doing just that, I don't get the point as you can't trust the draft tokens before they are validated, so you're still stuck waiting for the target model.

Just want to suggest: Ask an LLM about it! If you have access to a reasoning model like o3, I've found it to be very helpful.

I think this answer is as good as any of the human-generated ones in the thread so far, but the real power is that you can ask it follow-up questions. https://chatgpt.com/share/6894504f-4458-8008-a8c9-f371588259...

Re: Running GPT-OSS-120B at 500 tokens per second on Nvidia GPUs

#40

Post by mutkach »

> Inspired by GPUs, we parallelized this effort across multiple engineers. One engineer tried vLLM, another SGLang, and a third worked on TensorRT-LLM. We were able to quickly get TensorRT-LLM working, which was fortunate as it is usually the most performant inference framework for LLMs.

> TensorRT-LLM

It is usually the hardest to setup correctly and is often out of the date regarding the relevant architectures. It also requires compiling the model on the exact same hardware-drivers-libraries stack as your production environment which is a great pain in the rear end to say the least. Multimodal setups also been a disaster - at least for a while - when it was near-impossible to make it work even for mainstream models - like Multimodal Llamas. The big question is whether it's worth it, since when running the GPT-OSS-120B on H100 using vLLM is flawless in comparison - and the throughput stays at 130-140 t/s for a single H100. (It's also somewhat a clickbait of a title - I was expecting to see 500t/s for a single GPU, when in fact it's just a tensor-parallel setup)

It's also funny that they went for a separate release of TRT-LLM just to make sure that gpt-oss will work correctly, TRT-LLM is a mess

Post reply on HN