Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Running GPT-OSS-120B at 500 tokens per second on Nvidia GPUs

baseten.co

Post reply on HN
Print view View original

141–150 of 180 posts

Re: Running GPT-OSS-120B at 500 tokens per second on Nvidia GPUs

#141

Post by rpdillon »

rebolek wrote: post #136
beAbU wrote: post #95

Earlier quoted context omitted.

GPU stands for "graphics processing unit" so I'm not sure how your suggestion solves it. Maybe renaming the device to an MPU, where the M stands for "matrix/math/mips" would make it more semantically correct?

I think that G was changed to "general", so now it's "general processing unit".

This doesn't seem to be true at all. It's a highly specialized chip for doing highly parallel operations. There's nothing general about it.

I looked around briefly and could find no evidence that it's been renamed. Do you have a source?

Re: Running GPT-OSS-120B at 500 tokens per second on Nvidia GPUs

#142

Post by mikewarot »

You know what's actually hard to find in all this? The actual dimensions of the arrays in the model GPT-OSS-120B. At least with statically typed languages, you know how big your arrays are at a glance. I'm trying to find it in the GitHub repo[1], and I'm not seeing it.

I'm just trying to figure out how wide the datastream through this is, in particular, the actual data (not the weights) that flow through all of it. The width of the output stream. Just how big is a token at the output, prior to reducing it with "temperature" to a few bytes?

Assume infinitely fast compute in a magic black box, but you have to send the output through gigabit ethernet... what's the maximum number of tokens per second?

[1] https://github.com/openai/gpt-oss/tree/main/gpt_oss

Re: Running GPT-OSS-120B at 500 tokens per second on Nvidia GPUs

#143

Post by woleium »

radarsat1 wrote: post #131
wcallahan wrote: post #54

I just used GPT-OSS-120B on a cross Atlantic flight on my MacBook Pro (M4, 128GB RAM). A few things I noticed: - it’s only fast with with small context windows and small total token context; once more than ~10k tokens you’re basically queueing everything for a long time - MCPs/web search/url fetch have already become a very important part of interacting with LLMs; when they’re not available the LLM utility is greatly…

How long did your battery last?!

planes have power sockets now, but i do wonder how much jet fuel a whole plane of gpus would consume in electricity (assuming the system can handle it, which seems unlikely) and air conditioning.

Re: Running GPT-OSS-120B at 500 tokens per second on Nvidia GPUs

#144

Post by DrPhish »

wkat4242 wrote: post #58
DrPhish wrote: post #51

Earlier quoted context omitted.

2xEPYC Genoa w/768GB of DDR5-4800 and an A5000 24GB card. I built it in January 2024 for about $6k and have thoroughly enjoyed running every new model as it gets released. Some of the best money I’ve ever spent.

Wow nice!! That's a really good deal for that much hardware. How many tokens/s do you get for DeepSeek-R1?

Thanks, it was a bit of a gamble at the time (lots of dodgy ebay parts), but it paid off.

R1 starts at about 10t/s on an empty context but quickly falls off. I'd say the majority of my tokens are generating around 6t/s.

Some of the other big MoE models can be quite a bit faster.

I'm mostly using QwenCoder 480b at Q8 these days for 9t/s average. I've found I get better real-world results out of it than K2, R1 or GLM4.5.

Re: Running GPT-OSS-120B at 500 tokens per second on Nvidia GPUs

#145

Post by amluto »

mikewarot wrote: post #142

You know what's actually hard to find in all this? The actual dimensions of the arrays in the model GPT-OSS-120B. At least with statically typed languages, you know how big your arrays are at a glance. I'm trying to find it in the GitHub repo[1], and I'm not seeing it. I'm just trying to figure out how wide the datastream through this is, in particular, the actual data (not the weights) that flow through all of it. T…

What’s the application where you want to stream out the logits for each consecutive token while still sampling each token according to the usual rule? Keep in mind that, if you are doing the usual clever tricks like restricting the next token sampled to something that satisfies a grammar, you need to process the logits and sample them and return a token before running the next round of inference.

Re: Running GPT-OSS-120B at 500 tokens per second on Nvidia GPUs

#146

Post by eric-burel »

geertj wrote: post #46
eric-burel wrote: post #34

"Encourage Open-Source and Open-Weight AI" is the part just after "Ensure that Frontier AI Protects Free Speech and American Values" in America's AI Action Plan. I know this is not rational but OpenAI OSS models kinda give me chills as I am reading the Plan in parallel. Anyway I like seeing oss model providers talking about hardware, because that's a limiting point for most developers that are not familiar with this…

> Ensure that Frontier AI Protects Free Speech and American Values I am in the early phases of collecting my thoughts on this topic so bear with me, but it this a bad thing? AI models will have a world view. I think I prefer them having a western world view, as that has built our modern society and has proven to be most successful in making the lives of people better. At the very minimum I would want a model to docum…

Yeah I mean you'd want to take a look at the plan to get a bigger picture, it reflects a specific set of values which are not universally shared. This should led to the development of European models, but it feels inefficient to duplicate the work in each country/region just because open source models are planned to be used as trojan horses for values.

Re: Running GPT-OSS-120B at 500 tokens per second on Nvidia GPUs

#147

Post by reactordev »

diggan wrote: post #113
reactordev wrote: post #104

Earlier quoted context omitted.

huggingface has this built in if you care to fill out your software and hardware profile here: https://huggingface.co/settings/local-apps Then on the model pages, it will show you whether you can use it.

Interesting, never knew about that! I filled out my details, then went to https://huggingface.co/openai/gpt-oss-120b but I'm not sure if I see any difference? Where is it supposed to show if I can run it or not?

You’ll see green check next to models you can use on the model card.

https://huggingface.co/unsloth/gpt-oss-20b-GGUF

Re: Running GPT-OSS-120B at 500 tokens per second on Nvidia GPUs

#148

Post by mich5632 »

wcallahan wrote: post #54

I just used GPT-OSS-120B on a cross Atlantic flight on my MacBook Pro (M4, 128GB RAM). A few things I noticed: - it’s only fast with with small context windows and small total token context; once more than ~10k tokens you’re basically queueing everything for a long time - MCPs/web search/url fetch have already become a very important part of interacting with LLMs; when they’re not available the LLM utility is greatly…

I think this the difference between compute bound pre-fill (a cpu has a high bandwidth/compute ratio), vs decode. The time to first token is below 0.5s - even for a 10k context.

Re: Running GPT-OSS-120B at 500 tokens per second on Nvidia GPUs

#149

Post by diggan »

zackangelo wrote: post #138

GPT-OSS will run even faster on Blackwell chips because of its hardware support for fp4. If anyone is working on training or inference in Rust, I'm currently working on adding fp8 and fp4 support to cudarc[0] and candle[1]. This is being done so I can support these models in our inference engine for Mixlayer[2]. [0] https://github.com/coreylowman/cudarc/pull/449 [1] https://github.com/huggingface/candle/pull/2989 [2]…

Ah, interesting. As someone with a RTX Pro 6000, is it ready today to be able to run gpt-oss-120b inference, or are there still missing pieces? Both linked PRs seems merged already, so unsure if it's ready to be played around with or not.

Re: Running GPT-OSS-120B at 500 tokens per second on Nvidia GPUs

#150

Post by diggan »

mutkach wrote: post #41
acters wrote: post #18

Earlier quoted context omitted.

Personally, I think bigger companies should be more proactive and work with some of the popular inference engine software devs with getting their special snowflake LLM to work before it gets released. I guess it is all very much experimental at the end of the day. Those devs are putting in God's work for us to use on our budget friendly hardware choices.

This is a good take, actually. GPT-OSS is not much of a snowflake (judging by the model's architecture card at least) but TRT-LLM treats every model like that - there is too much hardcode - which makes it very difficult to just use it out-of-the-box for the hottest SotA thing.

> GPT-OSS is not much of a snowflake

Yeah, according to the architecture it doesn't seem like a snowflake, but they also decided to invent a new prompting/conversation format (https://github.com/openai/harmony) which definitely makes it a bit of a snowflake today, can't just use what worked a couple of days ago, but everyone needs to add proper support for it.

Post reply on HN