Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Running GPT-OSS-120B at 500 tokens per second on Nvidia GPUs

baseten.co

Post reply on HN
Print view View original

151–160 of 180 posts

Re: Running GPT-OSS-120B at 500 tokens per second on Nvidia GPUs

#151

Post by diggan »

lagrange77 wrote: post #129
lagrange77 wrote: post #91

While you're here.. Do you guys know a website that clearly shows which OS LLM models run on / fit into a specific GPU(setup)? The best heuristic i could find for the necessary VRAM is Number of Parameters × (Precision / 8) × 1.2 from here [0]. [0] https://medium.com/@lmpo/a-guide-to-estimating-vram-for-llms...

Thanks for your answers! While it is seemingly hard to calculate it, maybe one should just make a database website that tracks specific setups (model, exact variant / quantisation, runner, hardware) where users can report, which combination they got running (or not) along with metrics like tokens/s. Visitors could then specify their runner and hardware and filter for a list of models that would run on that.

Yeah, what you're suggesting sounds like it could be more useful than the "generalized calculators" people are currently publishing and using.

Re: Running GPT-OSS-120B at 500 tokens per second on Nvidia GPUs

#152

Post by diggan »

reactordev wrote: post #147
diggan wrote: post #113

Earlier quoted context omitted.

Interesting, never knew about that! I filled out my details, then went to https://huggingface.co/openai/gpt-oss-120b but I'm not sure if I see any difference? Where is it supposed to show if I can run it or not?

You’ll see green check next to models you can use on the model card. https://huggingface.co/unsloth/gpt-oss-20b-GGUF

Ah, it only works for GGUF, not for .safetensors (which the format HuggingFace themselves came up with :P ) ? I see the checks at https://huggingface.co/unsloth/gpt-oss-20b-GGUF but nothing at https://huggingface.co/openai/gpt-oss-120b, seems a bit backwards.

Re: Running GPT-OSS-120B at 500 tokens per second on Nvidia GPUs

#153

Post by fouc »

rebolek wrote: post #136
beAbU wrote: post #95

Earlier quoted context omitted.

GPU stands for "graphics processing unit" so I'm not sure how your suggestion solves it. Maybe renaming the device to an MPU, where the M stands for "matrix/math/mips" would make it more semantically correct?

I think that G was changed to "general", so now it's "general processing unit".

CPU is already the general (computing) processing unit so that wouldn't make sense

Re: Running GPT-OSS-120B at 500 tokens per second on Nvidia GPUs

#154

Post by fouc »

wcallahan wrote: post #54

I just used GPT-OSS-120B on a cross Atlantic flight on my MacBook Pro (M4, 128GB RAM). A few things I noticed: - it’s only fast with with small context windows and small total token context; once more than ~10k tokens you’re basically queueing everything for a long time - MCPs/web search/url fetch have already become a very important part of interacting with LLMs; when they’re not available the LLM utility is greatly…

What was your iogpu.wired_limit_mb set to? By default only ~70% or ~90GB of your RAM will be available to your GPU cores unless you change your wired limit setting.

Re: Running GPT-OSS-120B at 500 tokens per second on Nvidia GPUs

#155

Post by fouc »

DrPhish wrote: post #51
wkat4242 wrote: post #44

Earlier quoted context omitted.

What hardware do you have? 50tk/s is really impressive for cpu.

2xEPYC Genoa w/768GB of DDR5-4800 and an A5000 24GB card. I built it in January 2024 for about $6k and have thoroughly enjoyed running every new model as it gets released. Some of the best money I’ve ever spent.

I've seen some mentions of pure-cpu setups being successful for large models using old epyc/xeon workstations off ebay with 40+ cpus. Interesting approach!

Re: Running GPT-OSS-120B at 500 tokens per second on Nvidia GPUs

#156

Post by XCSme »

Aurornis wrote: post #115
XCSme wrote: post #84

Earlier quoted context omitted.

I know there was a downloadable version of Wikipedia (not that large). Maybe soon we'll have a lot of data stored locally and expose it via MCP, then the AIs can do "web search" locally. I think 99% of web searches lead to the same 100-1k websites. I assume it's only a few GBs to have a copy of those locally, thus this raises copyright concerns.

The mostly static knowledge content from sites like Wikipedia is already well represented in LLMs. LLMs call out to external websites when something isn’t commonly represented in training data, like specific project documentation or news events.

That's true, but the data is only approximately represented in the weights.

Maybe it's better to have the AI only "reason", and somehow instantly access precise data.

Re: Running GPT-OSS-120B at 500 tokens per second on Nvidia GPUs

#158

Post by adsharma »

XCSme wrote: post #156
Aurornis wrote: post #115

Earlier quoted context omitted.

The mostly static knowledge content from sites like Wikipedia is already well represented in LLMs. LLMs call out to external websites when something isn’t commonly represented in training data, like specific project documentation or news events.

That's true, but the data is only approximately represented in the weights. Maybe it's better to have the AI only "reason", and somehow instantly access precise data.

What use cases will gain from this architecture?

Re: Running GPT-OSS-120B at 500 tokens per second on Nvidia GPUs

#160

Post by mikewarot »

amluto wrote: post #145
mikewarot wrote: post #142

You know what's actually hard to find in all this? The actual dimensions of the arrays in the model GPT-OSS-120B. At least with statically typed languages, you know how big your arrays are at a glance. I'm trying to find it in the GitHub repo[1], and I'm not seeing it. I'm just trying to figure out how wide the datastream through this is, in particular, the actual data (not the weights) that flow through all of it. T…

What’s the application where you want to stream out the logits for each consecutive token while still sampling each token according to the usual rule? Keep in mind that, if you are doing the usual clever tricks like restricting the next token sampled to something that satisfies a grammar, you need to process the logits and sample them and return a token before running the next round of inference.

I know the actual output of the model is wider than a token.... but I can't find it (the actual width, or number of bytes) in the source. Perhaps it's my very casual familiarity with Python that's limiting me, but I don't see any actual declarations of array sizes anywhere in the code.

I'm just trying to calculate the actual bandwidth required for the full output of the model, not just a token to be handed off to the user.

I need this so I can compute just what bandwidth a fully FPGA (later ASIC) based implementation of the model would result in.

Edit/Append: I asked GPT-5, and it estimated:

  Total bytes = 50,000 tokens × 4 bytes/token = 200,000 bytes
Which sounds about right to me. This yields a maximum of about 500 logits/second on Gigabit ethernet.

The actual compute of the model is peanuts compared to just shuffling the data around.

Post reply on HN