TensorRT-LLM is a right nightmare to setup and maintain. Good on them for getting it to work for them - but it's not for everyone.
Running GPT-OSS-120B at 500 tokens per second on Nvidia GPUs
71–80 of 180 posts
Re: Running GPT-OSS-120B at 500 tokens per second on Nvidia GPUs
#72Re: Running GPT-OSS-120B at 500 tokens per second on Nvidia GPUs
#73"Encourage Open-Source and Open-Weight AI" is the part just after "Ensure that Frontier AI Protects Free Speech and American Values" in America's AI Action Plan. I know this is not rational but OpenAI OSS models kinda give me chills as I am reading the Plan in parallel. Anyway I like seeing oss model providers talking about hardware, because that's a limiting point for most developers that are not familiar with this…
> Ensure that Frontier AI Protects Free Speech and American Values I am in the early phases of collecting my thoughts on this topic so bear with me, but it this a bad thing? AI models will have a world view. I think I prefer them having a western world view, as that has built our modern society and has proven to be most successful in making the lives of people better. At the very minimum I would want a model to docum…
"Western" != "American": I grew up in a country where even the police are not, and do not wish to be, routinely armed.
Even then, there is an important difference between de-facto and de-jure rules. Fun fact: even North Korea has a constitutional guarantee of freedom of speech and the right vote*. They don't do these things as we would understand any of those words, but they have those things right there in the constitution.
So: does the USA, as it exists today, represent the values you want? Can you honestly say, hand on heart, that Alligator Alcatraz should be a thing your AI has been trained to support? Or that it's fine for Qatar to donate a 747 that becomes part of the library of the current president, not the office of the president, when his term in office comes to an end?
I won't list everything, this isn't the place for that, but even if we wind the clock back a few years, do you (/we) want an AI aligned with a political circus of kayfabe that distracts us from the real political machinations?
Of course, this is still USA-focused.
I'd say that what really made a difference to our quality of life wasn't even the American political system: there were massive improvements to human existence starting with the first industrial revolution in the UK in the 1760s, but the social and political nature of the world back then was so bleak that communism got invented a century later and introduced what was at the time controversial ideas like "women are not property" and "universal free education is good", and the USA's systems changed substantially several times since then (at a minimum Civil War, New Deal, and the Civil Rights movement).
The "meta system" that allows change can be considered good, but not uniquely so if you compare this to the Russian Revolution getting rid of the Tzars and a 40 years later they were in orbit (and this despite the Holodomor and WW2) and then threw off these shackles with Glasnost and the fall of the USSR (and note there that in Russia specifically, not all the former soviet countries but specifically Russia, the freedom gained failed to bring material improvements and the lives of those living through it were, in aggregate, made worse despite that freedom), and similar stories with the Chinese starting with dangerous incompetence (Four Pests campaign) and now in a position where "which is more powerful, them or the USA?" is a matter of which measure you use rather than it being obvious.
* https://en.wikipedia.org/wiki/Constitution_of_North_Korea#Ch...
Re: Running GPT-OSS-120B at 500 tokens per second on Nvidia GPUs
#74Earlier quoted context omitted.
Does it even make sense calling them 'GPUs' (I just checked NVIDIA product page for the H100 and it is indeed so)? There should be a quicker way to differentiate between 'consumer-grade hardware that is mainly meant to be used for gaming and can also run LLMs inference in a limited way' and 'business-grade hardware whose main purpose is AI training or running inference for LLMs".
We are fast approaching the return of the math coprocessor . In fashion they say that trends tend to reappear roughly every two decades, its overdue.
Yeah I would love for Nvidia to introduce faster update cycle to their hardware, so that we'll have models like "H201", "H220", etc.
I think it will also make sense to replace "H" with a brand number, sort of like they already do for customer GPUs.
So then maybe one day we'll have a math coprocessor called "Nvidia 80287".
Re: Running GPT-OSS-120B at 500 tokens per second on Nvidia GPUs
#75I just used GPT-OSS-120B on a cross Atlantic flight on my MacBook Pro (M4, 128GB RAM). A few things I noticed: - it’s only fast with with small context windows and small total token context; once more than ~10k tokens you’re basically queueing everything for a long time - MCPs/web search/url fetch have already become a very important part of interacting with LLMs; when they’re not available the LLM utility is greatly…
M2 Max processor.
I saw 60+ tok/s on short conversations, but it degraded to 30 tok/s as the conversation got longer.
Do you know what actually accounts for this slowdown? I don’t believe it was thermal throttling.
Re: Running GPT-OSS-120B at 500 tokens per second on Nvidia GPUs
#76I just used GPT-OSS-120B on a cross Atlantic flight on my MacBook Pro (M4, 128GB RAM). A few things I noticed: - it’s only fast with with small context windows and small total token context; once more than ~10k tokens you’re basically queueing everything for a long time - MCPs/web search/url fetch have already become a very important part of interacting with LLMs; when they’re not available the LLM utility is greatly…
Are you using Ollama or LMStudio/llama.cpp? https://x.com/ggerganov/status/1953088008816619637
Re: Running GPT-OSS-120B at 500 tokens per second on Nvidia GPUs
#77I just used GPT-OSS-120B on a cross Atlantic flight on my MacBook Pro (M4, 128GB RAM). A few things I noticed: - it’s only fast with with small context windows and small total token context; once more than ~10k tokens you’re basically queueing everything for a long time - MCPs/web search/url fetch have already become a very important part of interacting with LLMs; when they’re not available the LLM utility is greatly…
M2 Max processor. I saw 60+ tok/s on short conversations, but it degraded to 30 tok/s as the conversation got longer. Do you know what actually accounts for this slowdown? I don’t believe it was thermal throttling.
Physics: You always have the same memory bandwidth. The longer the context, the more bits will need to pass through the same pipe. Context is cumulative.
Re: Running GPT-OSS-120B at 500 tokens per second on Nvidia GPUs
#78Re: Running GPT-OSS-120B at 500 tokens per second on Nvidia GPUs
#79Earlier quoted context omitted.
What hardware do you have? 50tk/s is really impressive for cpu.
2xEPYC Genoa w/768GB of DDR5-4800 and an A5000 24GB card. I built it in January 2024 for about $6k and have thoroughly enjoyed running every new model as it gets released. Some of the best money I’ve ever spent.
thats a r/localllama user right there
Re: Running GPT-OSS-120B at 500 tokens per second on Nvidia GPUs
#80Earlier quoted context omitted.
M2 Max processor. I saw 60+ tok/s on short conversations, but it degraded to 30 tok/s as the conversation got longer. Do you know what actually accounts for this slowdown? I don’t believe it was thermal throttling.
Physics: You always have the same memory bandwidth. The longer the context, the more bits will need to pass through the same pipe. Context is cumulative.
No I don't think it's the bits. I would say it's the computation. Inference requires performing a lot of matmul, and with more tokens the number of computation operations increases exponentially - O(n^2) at least. So increasing your context/conversation will quickly degrade performance
I seriously doubt it's the throughput of memory during inference that's the bottleneck here.