Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Running GPT-OSS-120B at 500 tokens per second on Nvidia GPUs

baseten.co

Post reply on HN
Print view View original

61–70 of 180 posts

Re: Running GPT-OSS-120B at 500 tokens per second on Nvidia GPUs

#61

Post by amelius »

Kurtz79 wrote: post #59
blitzar wrote: post #55

> widely-available H100 GPUs Just looked in the parts drawer at home and dont seem to have a $25,000 GPU for some inexplicable reason.

Does it even make sense calling them 'GPUs' (I just checked NVIDIA product page for the H100 and it is indeed so)? There should be a quicker way to differentiate between 'consumer-grade hardware that is mainly meant to be used for gaming and can also run LLMs inference in a limited way' and 'business-grade hardware whose main purpose is AI training or running inference for LLMs".

Well, does it come with graphics connectors?

Re: Running GPT-OSS-120B at 500 tokens per second on Nvidia GPUs

#62

Post by blitzar »

Kurtz79 wrote: post #59
blitzar wrote: post #55

> widely-available H100 GPUs Just looked in the parts drawer at home and dont seem to have a $25,000 GPU for some inexplicable reason.

Does it even make sense calling them 'GPUs' (I just checked NVIDIA product page for the H100 and it is indeed so)? There should be a quicker way to differentiate between 'consumer-grade hardware that is mainly meant to be used for gaming and can also run LLMs inference in a limited way' and 'business-grade hardware whose main purpose is AI training or running inference for LLMs".

We are fast approaching the return of the math coprocessor. In fashion they say that trends tend to reappear roughly every two decades, its overdue.

Re: Running GPT-OSS-120B at 500 tokens per second on Nvidia GPUs

#63

Post by amelius »

sarthaksoni wrote: post #30

Reading this made me realize how easy it is to set up GPT-OSS 20B in comparison. I had it running on my Mac in five minutes, thanks to Llama.

Why is it hard to set up llms? You can just ask an llm to do it for you, no? If this relatively simple task is already too much for llms then what good are they?

Re: Running GPT-OSS-120B at 500 tokens per second on Nvidia GPUs

#64

Post by swexbe »

blitzar wrote: post #60
KolmogorovComp wrote: post #57

Earlier quoted context omitted.

available != cheap

available /əˈveɪləbl/ adjective: available able to be used or obtained; at someone's disposal

You can rent one from most cloud providers for a few bucks an hour.

Re: Running GPT-OSS-120B at 500 tokens per second on Nvidia GPUs

#66

Post by CraigRood »

sarthaksoni wrote: post #30

Reading this made me realize how easy it is to set up GPT-OSS 20B in comparison. I had it running on my Mac in five minutes, thanks to Llama.

I was playing with it yesterday and every single session gave me factually incorrect information.

Speed and ease of use is one thing, but it shouldn't be at the cost of accuracy.

Re: Running GPT-OSS-120B at 500 tokens per second on Nvidia GPUs

#67

Post by testaburger »

DrPhish wrote: post #51
wkat4242 wrote: post #44

Earlier quoted context omitted.

What hardware do you have? 50tk/s is really impressive for cpu.

2xEPYC Genoa w/768GB of DDR5-4800 and an A5000 24GB card. I built it in January 2024 for about $6k and have thoroughly enjoyed running every new model as it gets released. Some of the best money I’ve ever spent.

Which specific model epcys? And if it's not too much to ask which motherboard and power supply? I'm really interested in building something similar

Re: Running GPT-OSS-120B at 500 tokens per second on Nvidia GPUs

#68

Post by diggan »

amelius wrote: post #63
sarthaksoni wrote: post #30

Reading this made me realize how easy it is to set up GPT-OSS 20B in comparison. I had it running on my Mac in five minutes, thanks to Llama.

Why is it hard to set up llms? You can just ask an llm to do it for you, no? If this relatively simple task is already too much for llms then what good are they?

In the case of the GPT-OSS models, the worst (time consuming) part of supporting it is the new format they've trained the model with, "OpenAI harmony", in my own clients I couldn't just replace the model and call it a day, but still working on getting then to work correctly with tool calling...

Re: Running GPT-OSS-120B at 500 tokens per second on Nvidia GPUs

#69

Post by diggan »

acters wrote: post #18
tmshapland wrote: post #3

Such a fascinating read. I didn't realize how much massaging needed to be done to get the models to perform well. I just sort of assumed they worked out of the box.

Personally, I think bigger companies should be more proactive and work with some of the popular inference engine software devs with getting their special snowflake LLM to work before it gets released. I guess it is all very much experimental at the end of the day. Those devs are putting in God's work for us to use on our budget friendly hardware choices.

This is literally what they did for GPT-OSS, seems there was coordination to support it on day 1 with collaborations with OpenAI

Re: Running GPT-OSS-120B at 500 tokens per second on Nvidia GPUs

#70

Post by koakuma-chan »

swexbe wrote: post #64
blitzar wrote: post #60

Earlier quoted context omitted.

available /əˈveɪləbl/ adjective: available able to be used or obtained; at someone's disposal

You can rent one from most cloud providers for a few bucks an hour.

Might as well just use openai api
Post reply on HN