Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Running GPT-OSS-120B at 500 tokens per second on Nvidia GPUs

baseten.co

Post reply on HN
Print view View original

51–60 of 180 posts

Re: Running GPT-OSS-120B at 500 tokens per second on Nvidia GPUs

#51

Post by DrPhish »

wkat4242 wrote: post #44
DrPhish wrote: post #36

Earlier quoted context omitted.

Its also easy to do 120b on CPU if you have the resources. I had 120b running on my home LLM CPU inference box in just as long as it took to download the GGUFs, git pull and rebuild llama-server. I had it running at 40t/s with zero effort and 50t/s with a brief tweaking. Its just too bad that even the 120b isn't really worth running compared to the other models that are out there. It really is amazing what ggerganov…

What hardware do you have? 50tk/s is really impressive for cpu.

2xEPYC Genoa w/768GB of DDR5-4800 and an A5000 24GB card. I built it in January 2024 for about $6k and have thoroughly enjoyed running every new model as it gets released. Some of the best money I’ve ever spent.

Re: Running GPT-OSS-120B at 500 tokens per second on Nvidia GPUs

#52

Post by DrPhish »

exe34 wrote: post #49
DrPhish wrote: post #36

Earlier quoted context omitted.

Its also easy to do 120b on CPU if you have the resources. I had 120b running on my home LLM CPU inference box in just as long as it took to download the GGUFs, git pull and rebuild llama-server. I had it running at 40t/s with zero effort and 50t/s with a brief tweaking. Its just too bad that even the 120b isn't really worth running compared to the other models that are out there. It really is amazing what ggerganov…

I imagine the gguf is quantised stuff?

No, I’m running the unquantized 120b

Re: Running GPT-OSS-120B at 500 tokens per second on Nvidia GPUs

#53

Post by AesopAerial »

geertj wrote: post #46
eric-burel wrote: post #34

"Encourage Open-Source and Open-Weight AI" is the part just after "Ensure that Frontier AI Protects Free Speech and American Values" in America's AI Action Plan. I know this is not rational but OpenAI OSS models kinda give me chills as I am reading the Plan in parallel. Anyway I like seeing oss model providers talking about hardware, because that's a limiting point for most developers that are not familiar with this…

> Ensure that Frontier AI Protects Free Speech and American Values I am in the early phases of collecting my thoughts on this topic so bear with me, but it this a bad thing? AI models will have a world view. I think I prefer them having a western world view, as that has built our modern society and has proven to be most successful in making the lives of people better. At the very minimum I would want a model to docum…

> I think I prefer them having a western world view, as that has built our modern society and has proven to be most successful in making the lives of people better.

Highly debatable, and most people anywhere would probably say the same thing about whatever world view they hold.

Re: Running GPT-OSS-120B at 500 tokens per second on Nvidia GPUs

#54

Post by wcallahan »

I just used GPT-OSS-120B on a cross Atlantic flight on my MacBook Pro (M4, 128GB RAM).

A few things I noticed: - it’s only fast with with small context windows and small total token context; once more than ~10k tokens you’re basically queueing everything for a long time - MCPs/web search/url fetch have already become a very important part of interacting with LLMs; when they’re not available the LLM utility is greatly diminished - a lot of CLI/TUI coding tools (e.g., opencode) were not working reliably offline at this time with the model, despite being setup prior to being offline

That’s in addition to the other quirks others have noted with the OSS models.

Re: Running GPT-OSS-120B at 500 tokens per second on Nvidia GPUs

#56

Post by CMCDragonkai »

eric-burel wrote: post #31
acters wrote: post #18

Earlier quoted context omitted.

Personally, I think bigger companies should be more proactive and work with some of the popular inference engine software devs with getting their special snowflake LLM to work before it gets released. I guess it is all very much experimental at the end of the day. Those devs are putting in God's work for us to use on our budget friendly hardware choices.

SMEs are starting to want local LLMs and it's a nightmare to figure what hardware would work for what models. I am asking devs in my hometown to literally visit their installs to figure combos that work.

Are you installing them onsite?

Re: Running GPT-OSS-120B at 500 tokens per second on Nvidia GPUs

#58

Post by wkat4242 »

DrPhish wrote: post #51
wkat4242 wrote: post #44

Earlier quoted context omitted.

What hardware do you have? 50tk/s is really impressive for cpu.

2xEPYC Genoa w/768GB of DDR5-4800 and an A5000 24GB card. I built it in January 2024 for about $6k and have thoroughly enjoyed running every new model as it gets released. Some of the best money I’ve ever spent.

Wow nice!! That's a really good deal for that much hardware.

How many tokens/s do you get for DeepSeek-R1?

Re: Running GPT-OSS-120B at 500 tokens per second on Nvidia GPUs

#59

Post by Kurtz79 »

blitzar wrote: post #55

> widely-available H100 GPUs Just looked in the parts drawer at home and dont seem to have a $25,000 GPU for some inexplicable reason.

Does it even make sense calling them 'GPUs' (I just checked NVIDIA product page for the H100 and it is indeed so)?

There should be a quicker way to differentiate between 'consumer-grade hardware that is mainly meant to be used for gaming and can also run LLMs inference in a limited way' and 'business-grade hardware whose main purpose is AI training or running inference for LLMs".

Re: Running GPT-OSS-120B at 500 tokens per second on Nvidia GPUs

#60

Post by blitzar »

KolmogorovComp wrote: post #57
blitzar wrote: post #55

> widely-available H100 GPUs Just looked in the parts drawer at home and dont seem to have a $25,000 GPU for some inexplicable reason.

available != cheap

available /əˈveɪləbl/

adjective: available

able to be used or obtained; at someone's disposal

Post reply on HN