Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Running GPT-OSS-120B at 500 tokens per second on Nvidia GPUs

baseten.co

Post reply on HN
Print view View original

41–50 of 180 posts

Re: Running GPT-OSS-120B at 500 tokens per second on Nvidia GPUs

#41

Post by mutkach »

acters wrote: post #18
tmshapland wrote: post #3

Such a fascinating read. I didn't realize how much massaging needed to be done to get the models to perform well. I just sort of assumed they worked out of the box.

Personally, I think bigger companies should be more proactive and work with some of the popular inference engine software devs with getting their special snowflake LLM to work before it gets released. I guess it is all very much experimental at the end of the day. Those devs are putting in God's work for us to use on our budget friendly hardware choices.

This is a good take, actually. GPT-OSS is not much of a snowflake (judging by the model's architecture card at least) but TRT-LLM treats every model like that - there is too much hardcode - which makes it very difficult to just use it out-of-the-box for the hottest SotA thing.

Re: Running GPT-OSS-120B at 500 tokens per second on Nvidia GPUs

#43

Post by asabla »

modeless wrote: post #23
asabla wrote: post #14

Earlier quoted context omitted.

I'm on a 5090 so it's not apples to apples comparison. But I'm getting ~150t/s for the 20B version using ~16000 context size.

Cool, what software?

Initial testing has only been done with ollama. Plan on testing out llama.cpp and vllm when there is enough time

Re: Running GPT-OSS-120B at 500 tokens per second on Nvidia GPUs

#44

Post by wkat4242 »

DrPhish wrote: post #36
sarthaksoni wrote: post #30

Reading this made me realize how easy it is to set up GPT-OSS 20B in comparison. I had it running on my Mac in five minutes, thanks to Llama.

Its also easy to do 120b on CPU if you have the resources. I had 120b running on my home LLM CPU inference box in just as long as it took to download the GGUFs, git pull and rebuild llama-server. I had it running at 40t/s with zero effort and 50t/s with a brief tweaking. Its just too bad that even the 120b isn't really worth running compared to the other models that are out there. It really is amazing what ggerganov…

What hardware do you have? 50tk/s is really impressive for cpu.

Re: Running GPT-OSS-120B at 500 tokens per second on Nvidia GPUs

#45

Post by imtringued »

ahmedfromtunis wrote: post #9
furyofantares wrote: post #7

Earlier quoted context omitted.

Not an expert, but here's how I understand it. You know how input tokens are cheaper than output tokens? It's related to that. Say the model so far has "The capital of France". The small model generates "is Paris.", which let's say is 5 tokens. You feed the large model "The capital of France is Paris." to validate all 5 of those tokens in a single forward pass.

But what would happen if the small model's prediction was "is Rome."? Wouldn't that result in costlier inference if the small model is "wrong" more than it is correct. Also, if the small model would be sufficiently more "correct" than "wrong", wouldn't be more efficient to get rid of the large model at this point?

You're forgetting that some sequences are more predictable than others, hence the name "speculative" decoding. Let's say your token encoding has 128k tokens. That means the model has to pick the right token out of 128k. Some of those tokens are incredibly rare, while others are super common. The big model has seen the rare tokens many more times than the small model. This means that the small model will be able to do things like produce grammatically correct English, but not know anything about a specific JS framework.

The post training fine tuning costs (low thousand dollars) are the main reason why speculative decoding is relatively unpopular. The most effective speculative decoding strategy requires you to train multiple prediction heads ala medusa (or whatever succeeded it). If you don't do any fine tuning, then the probability of the small model being useful is slim. Using a random model as your draft model will probably give you very disappointing results.

Re: Running GPT-OSS-120B at 500 tokens per second on Nvidia GPUs

#46

Post by geertj »

eric-burel wrote: post #34

"Encourage Open-Source and Open-Weight AI" is the part just after "Ensure that Frontier AI Protects Free Speech and American Values" in America's AI Action Plan. I know this is not rational but OpenAI OSS models kinda give me chills as I am reading the Plan in parallel. Anyway I like seeing oss model providers talking about hardware, because that's a limiting point for most developers that are not familiar with this…

> Ensure that Frontier AI Protects Free Speech and American Values

I am in the early phases of collecting my thoughts on this topic so bear with me, but it this a bad thing?

AI models will have a world view. I think I prefer them having a western world view, as that has built our modern society and has proven to be most successful in making the lives of people better.

At the very minimum I would want a model to document its world view, and be aligned to it so that it does not try to socially engineer me to surreptitiously change mine.

Re: Running GPT-OSS-120B at 500 tokens per second on Nvidia GPUs

#48

Post by petesergeant »

geertj wrote: post #46
eric-burel wrote: post #34

"Encourage Open-Source and Open-Weight AI" is the part just after "Ensure that Frontier AI Protects Free Speech and American Values" in America's AI Action Plan. I know this is not rational but OpenAI OSS models kinda give me chills as I am reading the Plan in parallel. Anyway I like seeing oss model providers talking about hardware, because that's a limiting point for most developers that are not familiar with this…

> Ensure that Frontier AI Protects Free Speech and American Values I am in the early phases of collecting my thoughts on this topic so bear with me, but it this a bad thing? AI models will have a world view. I think I prefer them having a western world view, as that has built our modern society and has proven to be most successful in making the lives of people better. At the very minimum I would want a model to docum…

> but it this a bad thing?

I think the worry is that there’s no fixed definitions here, so the executive can use this to exert partisan or ideological pressure on model providers.

Every four years the models get RLHF’d to switch between thinking guns are amazing vs thinking guns are terrible.

Re: Running GPT-OSS-120B at 500 tokens per second on Nvidia GPUs

#49

Post by exe34 »

DrPhish wrote: post #36
sarthaksoni wrote: post #30

Reading this made me realize how easy it is to set up GPT-OSS 20B in comparison. I had it running on my Mac in five minutes, thanks to Llama.

Its also easy to do 120b on CPU if you have the resources. I had 120b running on my home LLM CPU inference box in just as long as it took to download the GGUFs, git pull and rebuild llama-server. I had it running at 40t/s with zero effort and 50t/s with a brief tweaking. Its just too bad that even the 120b isn't really worth running compared to the other models that are out there. It really is amazing what ggerganov…

I imagine the gguf is quantised stuff?

Re: Running GPT-OSS-120B at 500 tokens per second on Nvidia GPUs

#50

Post by exe34 »

geertj wrote: post #46
eric-burel wrote: post #34

"Encourage Open-Source and Open-Weight AI" is the part just after "Ensure that Frontier AI Protects Free Speech and American Values" in America's AI Action Plan. I know this is not rational but OpenAI OSS models kinda give me chills as I am reading the Plan in parallel. Anyway I like seeing oss model providers talking about hardware, because that's a limiting point for most developers that are not familiar with this…

> Ensure that Frontier AI Protects Free Speech and American Values I am in the early phases of collecting my thoughts on this topic so bear with me, but it this a bad thing? AI models will have a world view. I think I prefer them having a western world view, as that has built our modern society and has proven to be most successful in making the lives of people better. At the very minimum I would want a model to docum…

> I think I prefer them having a western world view,

What worries me is that the current "western world view" of America is not the same as the western world view we've shared with them since the cold war. The trend is towards the same kind of values and behaviour we see in the Islamic Republic and the Russian Federation. If that sort of "western world view" gets baked into the intelligent infrastructure, it may be very hard to change course in the future. For example dissidence and wrongthink is going to get harder and harder.

Post reply on HN