Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Running GPT-OSS-120B at 500 tokens per second on Nvidia GPUs

baseten.co

Post reply on HN
Print view View original

121–130 of 180 posts

Re: Running GPT-OSS-120B at 500 tokens per second on Nvidia GPUs

#121

Post by philipkiely »

nektro wrote: post #90

> we were the clear leader running on NVIDIA GPUs for both latency and throughput per public data from real-world use on OpenRouter. Baseten: 592.6 tps Groq: 784.6 tps Cerebras: 4,245 tps still impressive work

Yeah the custom hardware providers are super good at TPS. Kudos to their teams for sure, and the demos of instant reasoning are incredibly impressive.

That said, we are serving the model at its full 131K context window, and they are serving 33K max, which could matter for some edge case prompts.

Additionally, NVIDIA hardware is much more widely available if you are scaling a high-traffic application.

Re: Running GPT-OSS-120B at 500 tokens per second on Nvidia GPUs

#122

Post by Aurornis »

zackify wrote: post #110
wcallahan wrote: post #54

I just used GPT-OSS-120B on a cross Atlantic flight on my MacBook Pro (M4, 128GB RAM). A few things I noticed: - it’s only fast with with small context windows and small total token context; once more than ~10k tokens you’re basically queueing everything for a long time - MCPs/web search/url fetch have already become a very important part of interacting with LLMs; when they’re not available the LLM utility is greatly…

You didn’t even mention how it’ll be on fire unless you use low power mode. Yes all this has been known since the M4 came out. The memory bandwidth is too low. Try using it with real tasks like cline or opencode and the context length is too long and slow to be practical

> Yes all this has been known since the M4 came out. The memory bandwidth is too low.

The M4 Max with 128GB of RAM (the part used in the comment) has over 500GB/sec of memory bandwidth.

Re: Running GPT-OSS-120B at 500 tokens per second on Nvidia GPUs

#123

Post by philipkiely »

smcleod wrote: post #71

TensorRT-LLM is a right nightmare to setup and maintain. Good on them for getting it to work for them - but it's not for everyone.

We have built a ton of tooling on top of TRT-LLM and use it not just for LLMs but also for TTS models (Orpheus), STT models (Whisper), and embedding models.

Re: Running GPT-OSS-120B at 500 tokens per second on Nvidia GPUs

#124

Post by varispeed »

Aurornis wrote: post #118
blitzar wrote: post #55

> widely-available H100 GPUs Just looked in the parts drawer at home and dont seem to have a $25,000 GPU for some inexplicable reason.

They’re widely available to rent. Unless you’re running it 24/7 for multiple years, it’s not going to be cost effective to buy the GPU instead of renting a hosted one. For personal use you wouldn’t get a recent generation data center card anyway. You’d get something like a Mac Studio or Strix Halo and deal with the slower speed.

I rented H100 for training a couple of times and I found that they couldn't do training at all. Same code worked fine on Mac M1 or RTX 5080, but on H100 I was getting completely different results.

So I wonder what I could be doing wrong. In the end I just use RTX 5080 as my models fit neatly in the available RAM.

* by not working at all, I mean the scripts worked, but results were wrong. As if H100 couldn't do maths properly.

Re: Running GPT-OSS-120B at 500 tokens per second on Nvidia GPUs

#125

Post by torginus »

MoonObserver wrote: post #75
wcallahan wrote: post #54

I just used GPT-OSS-120B on a cross Atlantic flight on my MacBook Pro (M4, 128GB RAM). A few things I noticed: - it’s only fast with with small context windows and small total token context; once more than ~10k tokens you’re basically queueing everything for a long time - MCPs/web search/url fetch have already become a very important part of interacting with LLMs; when they’re not available the LLM utility is greatly…

M2 Max processor. I saw 60+ tok/s on short conversations, but it degraded to 30 tok/s as the conversation got longer. Do you know what actually accounts for this slowdown? I don’t believe it was thermal throttling.

Inference takes quadratic amount of time wrt context size

Re: Running GPT-OSS-120B at 500 tokens per second on Nvidia GPUs

#126

Post by esafak »

diggan wrote: post #100
conradev wrote: post #76

Earlier quoted context omitted.

Are you using Ollama or LMStudio/llama.cpp? https://x.com/ggerganov/status/1953088008816619637

> LMStudio/llama.cpp Even though LM Studio uses llama.cpp as a runtime, the performance differs between them. With LM Studio 0.3.22 Build 2 with CUDA Llama.cpp (Linux) v1.45.0 runtime I get ~86 tok/s on a RTX Pro 6000, while with llama.cpp compiled from 1d72c841888 (Aug 7 10:53:21 2025) I get ~180 tok/s, almost 100 more per second, both running lmstudio-community/gpt-oss-120b-GGUF.

Is it always like this or does it depend on the model?

Re: Running GPT-OSS-120B at 500 tokens per second on Nvidia GPUs

#127

Post by esafak »

MereInterest wrote: post #99
VierScar wrote: post #80

Earlier quoted context omitted.

No I don't think it's the bits. I would say it's the computation. Inference requires performing a lot of matmul, and with more tokens the number of computation operations increases exponentially - O(n^2) at least. So increasing your context/conversation will quickly degrade performance I seriously doubt it's the throughput of memory during inference that's the bottleneck here.

Nitpick: O(n^2) is quadratic, not exponential. For it to “increase exponentially”, n would need to be in the exponent, such as O(2^n).

To contrast with exponential, the term is power law.

Re: Running GPT-OSS-120B at 500 tokens per second on Nvidia GPUs

#128

Post by diggan »

esafak wrote: post #126
diggan wrote: post #100

Earlier quoted context omitted.

> LMStudio/llama.cpp Even though LM Studio uses llama.cpp as a runtime, the performance differs between them. With LM Studio 0.3.22 Build 2 with CUDA Llama.cpp (Linux) v1.45.0 runtime I get ~86 tok/s on a RTX Pro 6000, while with llama.cpp compiled from 1d72c841888 (Aug 7 10:53:21 2025) I get ~180 tok/s, almost 100 more per second, both running lmstudio-community/gpt-oss-120b-GGUF.

Is it always like this or does it depend on the model?

Depends on the model. Each runner needs to implement support when there are new architectures, and they all seemingly focuses on different things. As far as I've gathered so far, vLLM focuses on inference speed, SGLang on parallizing across multiple GPUs, Ollama on being as fast out the door with their implementation as possible, sometimes cutting corners, llama.cpp sits somewhere in-between Ollama and vLLM. Then LM Studio seems to lag slightly behind with their llama.cpp usage, so I'm guessing that's the difference between LM Studio and building llama.cpp from source today.

Re: Running GPT-OSS-120B at 500 tokens per second on Nvidia GPUs

#129

Post by lagrange77 »

lagrange77 wrote: post #91

While you're here.. Do you guys know a website that clearly shows which OS LLM models run on / fit into a specific GPU(setup)? The best heuristic i could find for the necessary VRAM is Number of Parameters × (Precision / 8) × 1.2 from here [0]. [0] https://medium.com/@lmpo/a-guide-to-estimating-vram-for-llms...

Thanks for your answers!

While it is seemingly hard to calculate it, maybe one should just make a database website that tracks specific setups (model, exact variant / quantisation, runner, hardware) where users can report, which combination they got running (or not) along with metrics like tokens/s.

Visitors could then specify their runner and hardware and filter for a list of models that would run on that.

Post reply on HN