Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Running GPT-OSS-120B at 500 tokens per second on Nvidia GPUs

baseten.co

Post reply on HN
Print view View original

111–120 of 180 posts

Re: Running GPT-OSS-120B at 500 tokens per second on Nvidia GPUs

#111

Post by sigbottle »

omneity wrote: post #107
codedokode wrote: post #102

Earlier quoted context omitted.

By the way I wonder, what has more performance, a $25 000 professional GPU or a bunch of cheaper consumer GPUs costing $25 000 in total?

Consumer GPUs in theory and by a large margin (10 5090s will eat an H100 lunch with 6 times the bandwidth, 3x VRAM and a relatively similar compute ratio), but your bottleneck is the interconnect and that is intentionally crippled to avoid beowulf GPU clusters eating into their datacenter market. Last consumer GPU with NVLink was the RTX 3090. Even the workstation-grade GPUs lost it. https://forums.developer.nvidia.c…

H100s also has custom async WGMMA instructions among other things. From what I understand, at least the async instructions formalize the notion of pipelining, which engineers were already implicitly using because to optimize memory accesses you're effectively trying to overlap them in that kind of optimal parallel manner.

Re: Running GPT-OSS-120B at 500 tokens per second on Nvidia GPUs

#112

Post by [deleted] »

beAbU wrote: post #95
addandsubtract wrote: post #92

Earlier quoted context omitted.

We could call the consumer ones GFX cards, and keep GPU for the matrix multiplying ones.

GPU stands for "graphics processing unit" so I'm not sure how your suggestion solves it. Maybe renaming the device to an MPU, where the M stands for "matrix/math/mips" would make it more semantically correct?

[deleted]

Re: Running GPT-OSS-120B at 500 tokens per second on Nvidia GPUs

#113

Post by diggan »

reactordev wrote: post #104
lagrange77 wrote: post #91

While you're here.. Do you guys know a website that clearly shows which OS LLM models run on / fit into a specific GPU(setup)? The best heuristic i could find for the necessary VRAM is Number of Parameters × (Precision / 8) × 1.2 from here [0]. [0] https://medium.com/@lmpo/a-guide-to-estimating-vram-for-llms...

huggingface has this built in if you care to fill out your software and hardware profile here: https://huggingface.co/settings/local-apps Then on the model pages, it will show you whether you can use it.

Interesting, never knew about that! I filled out my details, then went to https://huggingface.co/openai/gpt-oss-120b but I'm not sure if I see any difference? Where is it supposed to show if I can run it or not?

Re: Running GPT-OSS-120B at 500 tokens per second on Nvidia GPUs

#115

Post by Aurornis »

XCSme wrote: post #84
wcallahan wrote: post #54

I just used GPT-OSS-120B on a cross Atlantic flight on my MacBook Pro (M4, 128GB RAM). A few things I noticed: - it’s only fast with with small context windows and small total token context; once more than ~10k tokens you’re basically queueing everything for a long time - MCPs/web search/url fetch have already become a very important part of interacting with LLMs; when they’re not available the LLM utility is greatly…

I know there was a downloadable version of Wikipedia (not that large). Maybe soon we'll have a lot of data stored locally and expose it via MCP, then the AIs can do "web search" locally. I think 99% of web searches lead to the same 100-1k websites. I assume it's only a few GBs to have a copy of those locally, thus this raises copyright concerns.

The mostly static knowledge content from sites like Wikipedia is already well represented in LLMs.

LLMs call out to external websites when something isn’t commonly represented in training data, like specific project documentation or news events.

Re: Running GPT-OSS-120B at 500 tokens per second on Nvidia GPUs

#116

Post by philipkiely »

blitzar wrote: post #55

> widely-available H100 GPUs Just looked in the parts drawer at home and dont seem to have a $25,000 GPU for some inexplicable reason.

This comment made my day ty! Yeah definitely speaking from a datacenter perspective -- fastest piece of hardware I have in the parts drawer is probably my old iPhone 8.

Re: Running GPT-OSS-120B at 500 tokens per second on Nvidia GPUs

#117

Post by philipkiely »

mutkach wrote: post #40

> Inspired by GPUs, we parallelized this effort across multiple engineers. One engineer tried vLLM, another SGLang, and a third worked on TensorRT-LLM. We were able to quickly get TensorRT-LLM working, which was fortunate as it is usually the most performant inference framework for LLMs. > TensorRT-LLM It is usually the hardest to setup correctly and is often out of the date regarding the relevant architectures. It a…

TRT-LLM has its challenges from a DX perspective and yeah for Multi-modal we still use vLLM pretty often.

But for the kind of traffic we are trying to serve -- high volume and latency sensitive -- it consistently wins head-to-head in our benchmarking and we have invested a ton of dev work in the tooling around it.

Re: Running GPT-OSS-120B at 500 tokens per second on Nvidia GPUs

#118

Post by Aurornis »

blitzar wrote: post #55

> widely-available H100 GPUs Just looked in the parts drawer at home and dont seem to have a $25,000 GPU for some inexplicable reason.

They’re widely available to rent.

Unless you’re running it 24/7 for multiple years, it’s not going to be cost effective to buy the GPU instead of renting a hosted one.

For personal use you wouldn’t get a recent generation data center card anyway. You’d get something like a Mac Studio or Strix Halo and deal with the slower speed.

Re: Running GPT-OSS-120B at 500 tokens per second on Nvidia GPUs

#119

Post by philipkiely »

lagrange77 wrote: post #91

While you're here.. Do you guys know a website that clearly shows which OS LLM models run on / fit into a specific GPU(setup)? The best heuristic i could find for the necessary VRAM is Number of Parameters × (Precision / 8) × 1.2 from here [0]. [0] https://medium.com/@lmpo/a-guide-to-estimating-vram-for-llms...

Yeah we have tried to build calculators before it just depends so much.

Your equation is roughly correct, but I tend to multiply by a factor of 2 not 1.2 to allow for highly concurrent traffic.

Re: Running GPT-OSS-120B at 500 tokens per second on Nvidia GPUs

#120

Post by Aurornis »

dougSF70 wrote: post #85
blitzar wrote: post #55

> widely-available H100 GPUs Just looked in the parts drawer at home and dont seem to have a $25,000 GPU for some inexplicable reason.

With Ollama i got the 20B model running on 8 TitanX cards (2015). Ollama distributed the model so that the 15GB of vram required was split evenly accross the 8 cards. The tok/s were faster than reading speed.

For the price of 8 decade old Titan X cards, someone could pick up a single modern GPU with 16GB or more of RAM.
Post reply on HN