Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Running GPT-OSS-120B at 500 tokens per second on Nvidia GPUs

baseten.co

Post reply on HN
Print view View original

101–110 of 180 posts

Re: Running GPT-OSS-120B at 500 tokens per second on Nvidia GPUs

#101

Post by diggan »

lagrange77 wrote: post #91

While you're here.. Do you guys know a website that clearly shows which OS LLM models run on / fit into a specific GPU(setup)? The best heuristic i could find for the necessary VRAM is Number of Parameters × (Precision / 8) × 1.2 from here [0]. [0] https://medium.com/@lmpo/a-guide-to-estimating-vram-for-llms...

Maybe I'm spoiled by having great internet connection, but I usually download the weights and try to run them via various tools (llama.cpp, LM Studio, vLLM and SGLang typically) and see what works. There seems to be so many variables involved (runners, architectures, implementations, hardware and so on) that none of the calculators I've tried so far been accurate, both in the way that they've over-estimated and under-estimated what I could run.

So in the end, trying to actually run them seems to be the only fool-proof way of knowing for sure :)

Re: Running GPT-OSS-120B at 500 tokens per second on Nvidia GPUs

#102

Post by codedokode »

Kurtz79 wrote: post #59
blitzar wrote: post #55

> widely-available H100 GPUs Just looked in the parts drawer at home and dont seem to have a $25,000 GPU for some inexplicable reason.

Does it even make sense calling them 'GPUs' (I just checked NVIDIA product page for the H100 and it is indeed so)? There should be a quicker way to differentiate between 'consumer-grade hardware that is mainly meant to be used for gaming and can also run LLMs inference in a limited way' and 'business-grade hardware whose main purpose is AI training or running inference for LLMs".

By the way I wonder, what has more performance, a $25 000 professional GPU or a bunch of cheaper consumer GPUs costing $25 000 in total?

Re: Running GPT-OSS-120B at 500 tokens per second on Nvidia GPUs

#103

Post by diggan »

steinvakt2 wrote: post #27
asabla wrote: post #14

Earlier quoted context omitted.

I'm on a 5090 so it's not apples to apples comparison. But I'm getting ~150t/s for the 20B version using ~16000 context size.

And flash attention doesn't work on 5090 yet, right? So currently 4090 is probably faster, or?

> And flash attention doesn't work on 5090 yet, right?

Flash attention works with GPT-OSS + llama.cpp (tested on 1d72c8418) and other Blackwell card (RTX Pro 6000) so I think it should work on 5090 as well, it's the same architecture after all.

Re: Running GPT-OSS-120B at 500 tokens per second on Nvidia GPUs

#104

Post by reactordev »

lagrange77 wrote: post #91

While you're here.. Do you guys know a website that clearly shows which OS LLM models run on / fit into a specific GPU(setup)? The best heuristic i could find for the necessary VRAM is Number of Parameters × (Precision / 8) × 1.2 from here [0]. [0] https://medium.com/@lmpo/a-guide-to-estimating-vram-for-llms...

huggingface has this built in if you care to fill out your software and hardware profile here:

https://huggingface.co/settings/local-apps

Then on the model pages, it will show you whether you can use it.

Re: Running GPT-OSS-120B at 500 tokens per second on Nvidia GPUs

#105

Post by WithinReason »

blitzar wrote: post #62
Kurtz79 wrote: post #59

Earlier quoted context omitted.

Does it even make sense calling them 'GPUs' (I just checked NVIDIA product page for the H100 and it is indeed so)? There should be a quicker way to differentiate between 'consumer-grade hardware that is mainly meant to be used for gaming and can also run LLMs inference in a limited way' and 'business-grade hardware whose main purpose is AI training or running inference for LLMs".

We are fast approaching the return of the math coprocessor . In fashion they say that trends tend to reappear roughly every two decades, its overdue.

It's called a tensorcore and it's in most GPUs

Re: Running GPT-OSS-120B at 500 tokens per second on Nvidia GPUs

#106

Post by simpleintheory »

diggan wrote: post #96
OliverGuy wrote: post #88

Earlier quoted context omitted.

Nope, doesn't have any of the required hardware to even process graphics iirc

Although the RTX Pro 6000 is not consumer-grade, it does come with graphics ports (four Displayports) and does render graphics like a consumer card :) So seems the difference between the segments is becoming smaller, not bigger.

That’s because it’s intended as a workstation GPU not one used in servers

Re: Running GPT-OSS-120B at 500 tokens per second on Nvidia GPUs

#107

Post by omneity »

codedokode wrote: post #102
Kurtz79 wrote: post #59

Earlier quoted context omitted.

Does it even make sense calling them 'GPUs' (I just checked NVIDIA product page for the H100 and it is indeed so)? There should be a quicker way to differentiate between 'consumer-grade hardware that is mainly meant to be used for gaming and can also run LLMs inference in a limited way' and 'business-grade hardware whose main purpose is AI training or running inference for LLMs".

By the way I wonder, what has more performance, a $25 000 professional GPU or a bunch of cheaper consumer GPUs costing $25 000 in total?

Consumer GPUs in theory and by a large margin (10 5090s will eat an H100 lunch with 6 times the bandwidth, 3x VRAM and a relatively similar compute ratio), but your bottleneck is the interconnect and that is intentionally crippled to avoid beowulf GPU clusters eating into their datacenter market.

Last consumer GPU with NVLink was the RTX 3090. Even the workstation-grade GPUs lost it.

https://forums.developer.nvidia.com/t/rtx-a6000-ada-no-more-...

Re: Running GPT-OSS-120B at 500 tokens per second on Nvidia GPUs

#108

Post by diggan »

simpleintheory wrote: post #106
diggan wrote: post #96

Earlier quoted context omitted.

Although the RTX Pro 6000 is not consumer-grade, it does come with graphics ports (four Displayports) and does render graphics like a consumer card :) So seems the difference between the segments is becoming smaller, not bigger.

That’s because it’s intended as a workstation GPU not one used in servers

Sure, but it still sits in the 'business-grade hardware whose main purpose is AI training or running inference for LLMs" segment parent mentioned, yet have graphics connectors so the only thing I'm saying is that just looking at that won't help you understand what segment the GPU goes into.

Re: Running GPT-OSS-120B at 500 tokens per second on Nvidia GPUs

#109

Post by zozbot234 »

summarity wrote: post #93
VierScar wrote: post #80

Earlier quoted context omitted.

No I don't think it's the bits. I would say it's the computation. Inference requires performing a lot of matmul, and with more tokens the number of computation operations increases exponentially - O(n^2) at least. So increasing your context/conversation will quickly degrade performance I seriously doubt it's the throughput of memory during inference that's the bottleneck here.

It literally is. LLM inference is almost entirely memory bound. In fact for naive inference (no batching), you can calculate the token throughput just based on the model size, context size and memory bandwidth.

Prompt pre-processing (before the first token is output) is raw compute-bound. That's why it would be nice if we could direct llama.cpp/ollama to run that phase only on iGPU/NPU (for systems without a separate dGPU, obviously) and shift the whole thing over to CPU inference for the latter token-generation phase.

(A memory-bound workload like token gen wouldn't usually run into the CPU's thermal or power limits, so there would be little or no gain from offloading work to the iGPU/NPU in that phase.)

Re: Running GPT-OSS-120B at 500 tokens per second on Nvidia GPUs

#110

Post by zackify »

wcallahan wrote: post #54

I just used GPT-OSS-120B on a cross Atlantic flight on my MacBook Pro (M4, 128GB RAM). A few things I noticed: - it’s only fast with with small context windows and small total token context; once more than ~10k tokens you’re basically queueing everything for a long time - MCPs/web search/url fetch have already become a very important part of interacting with LLMs; when they’re not available the LLM utility is greatly…

You didn’t even mention how it’ll be on fire unless you use low power mode.

Yes all this has been known since the M4 came out. The memory bandwidth is too low.

Try using it with real tasks like cline or opencode and the context length is too long and slow to be practical

Post reply on HN