Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Running GPT-OSS-120B at 500 tokens per second on Nvidia GPUs

baseten.co

Post reply on HN
Print view View original

81–90 of 180 posts

Re: Running GPT-OSS-120B at 500 tokens per second on Nvidia GPUs

#81

Post by genewitch »

Kurtz79 wrote: post #59
blitzar wrote: post #55

> widely-available H100 GPUs Just looked in the parts drawer at home and dont seem to have a $25,000 GPU for some inexplicable reason.

Does it even make sense calling them 'GPUs' (I just checked NVIDIA product page for the H100 and it is indeed so)? There should be a quicker way to differentiate between 'consumer-grade hardware that is mainly meant to be used for gaming and can also run LLMs inference in a limited way' and 'business-grade hardware whose main purpose is AI training or running inference for LLMs".

"GPGPU" was something from over a decade ago; for general purpose GPU computing

Re: Running GPT-OSS-120B at 500 tokens per second on Nvidia GPUs

#82

Post by zozbot234 »

VierScar wrote: post #80
summarity wrote: post #77

Earlier quoted context omitted.

Physics: You always have the same memory bandwidth. The longer the context, the more bits will need to pass through the same pipe. Context is cumulative.

No I don't think it's the bits. I would say it's the computation. Inference requires performing a lot of matmul, and with more tokens the number of computation operations increases exponentially - O(n^2) at least. So increasing your context/conversation will quickly degrade performance I seriously doubt it's the throughput of memory during inference that's the bottleneck here.

Typically, the token generation phase is memory-bound for LLM inference in general, and this becomes especially clear as context length increases (since the model's parameters are a fixed quantity.) If it was pure compute bound there would be huge gains to be had by shifting some of the load to the NPU (ANE) but AIUI it's just not so.

Re: Running GPT-OSS-120B at 500 tokens per second on Nvidia GPUs

#83

Post by smartbit »

testaburger wrote: post #67
DrPhish wrote: post #51

Earlier quoted context omitted.

2xEPYC Genoa w/768GB of DDR5-4800 and an A5000 24GB card. I built it in January 2024 for about $6k and have thoroughly enjoyed running every new model as it gets released. Some of the best money I’ve ever spent.

Which specific model epcys? And if it's not too much to ask which motherboard and power supply? I'm really interested in building something similar

Looking at https://news.ycombinator.com/submitted?id=DrPhish it's probably this machine https://rentry.co/miqumaxx

  * Gigabyte MZ73-LM1 with two AMD EPYC GENOA 9334 QS 64c/128t
  * 24 sticks of M321R4GA3BB6-CQK 32GB DDR5-4800 RDIMM PC5-38400R
  * 24GB A5000
Note that the RAM price almost doubled since Jan 2024

Re: Running GPT-OSS-120B at 500 tokens per second on Nvidia GPUs

#84

Post by XCSme »

wcallahan wrote: post #54

I just used GPT-OSS-120B on a cross Atlantic flight on my MacBook Pro (M4, 128GB RAM). A few things I noticed: - it’s only fast with with small context windows and small total token context; once more than ~10k tokens you’re basically queueing everything for a long time - MCPs/web search/url fetch have already become a very important part of interacting with LLMs; when they’re not available the LLM utility is greatly…

I know there was a downloadable version of Wikipedia (not that large). Maybe soon we'll have a lot of data stored locally and expose it via MCP, then the AIs can do "web search" locally.

I think 99% of web searches lead to the same 100-1k websites. I assume it's only a few GBs to have a copy of those locally, thus this raises copyright concerns.

Re: Running GPT-OSS-120B at 500 tokens per second on Nvidia GPUs

#85

Post by dougSF70 »

blitzar wrote: post #55

> widely-available H100 GPUs Just looked in the parts drawer at home and dont seem to have a $25,000 GPU for some inexplicable reason.

With Ollama i got the 20B model running on 8 TitanX cards (2015). Ollama distributed the model so that the 15GB of vram required was split evenly accross the 8 cards. The tok/s were faster than reading speed.

Re: Running GPT-OSS-120B at 500 tokens per second on Nvidia GPUs

#86

Post by gigatexal »

wcallahan wrote: post #54

I just used GPT-OSS-120B on a cross Atlantic flight on my MacBook Pro (M4, 128GB RAM). A few things I noticed: - it’s only fast with with small context windows and small total token context; once more than ~10k tokens you’re basically queueing everything for a long time - MCPs/web search/url fetch have already become a very important part of interacting with LLMs; when they’re not available the LLM utility is greatly…

M3 Max 128GB here and it’s mad impressive.

Im spec’ing out a Mac Studio with 512GB ram because I can window shop and wish but I think the trend for local LLMs is getting really good.

Do we know WHY openAI even released them?

Re: Running GPT-OSS-120B at 500 tokens per second on Nvidia GPUs

#88

Post by OliverGuy »

amelius wrote: post #61
Kurtz79 wrote: post #59

Earlier quoted context omitted.

Does it even make sense calling them 'GPUs' (I just checked NVIDIA product page for the H100 and it is indeed so)? There should be a quicker way to differentiate between 'consumer-grade hardware that is mainly meant to be used for gaming and can also run LLMs inference in a limited way' and 'business-grade hardware whose main purpose is AI training or running inference for LLMs".

Well, does it come with graphics connectors?

Nope, doesn't have any of the required hardware to even process graphics iirc

Re: Running GPT-OSS-120B at 500 tokens per second on Nvidia GPUs

#89

Post by OliverGuy »

CraigRood wrote: post #66
sarthaksoni wrote: post #30

Reading this made me realize how easy it is to set up GPT-OSS 20B in comparison. I had it running on my Mac in five minutes, thanks to Llama.

I was playing with it yesterday and every single session gave me factually incorrect information. Speed and ease of use is one thing, but it shouldn't be at the cost of accuracy.

If you are trying to get facts out of an LLM you are using it wrong, if you want a fact it should use a tool (eg we search, rag etc) to get the information that contains the fact (Wikipedia page, documentation etc) and then parse that document for the fact and return it to you.
Post reply on HN