231–240 of 909 posts
Looks like Groq (at 1k+ tokens/second) and Fireworks are already live on openrouter: https://openrouter.ai/openai/gpt-oss-120b $0.15M in / $0.6-0.75M out edit: Now Cerebras too at 3,815 tps for $0.25M / $0.69M out.
Earlier quoted context omitted.
When people talk about running a (quantized) medium-sized model on a Mac Mini, what types of latency and throughput times are they talking about? Do they mean like 5 tokens per second or at an actually usable speed?
On a M1 MacBook Air with 8GB, I got this running Gemma 3n: 12.63 tok/sec • 860 tokens • 1.52s to first token I'm amazed it works at all with such limited RAM
Earlier quoted context omitted.
in that era, OpenAI and Anthropic were still deluding themselves into thinking they would be the "stewards" of generative AI, and the last US administration was very keen on regoolating everything under the sun, so "safety" was just an angle for regulatory capture. God bless China.
Oh absolutely, AI labs certainly talk their books, including any safety angles. The controversy/outrage extended far beyond those incentivized companies too. Many people had good faith worries about Llama. Open-weight models are now vastly more powerful than Llama-1, yet the sky hasn't fallen. It's just fascinating to me how apocalyptic people are. I just feel lucky to be around in what's likely the most important de…
Lol. To be young and foolish again. This covid laced decade is more of a placeholder. The current decade is always the most meaningful until the next one. The personal computer era, the first cars or planes, ending slavery needs to take a backseat to the best search engine ever. We are at the point where everyone is planning on what they are going to do with their hoverboards.
Why do companies release open source LLMs? I would understand it, if there was some technology lock-in. But with LLMs, there is no such thing. One can switch out LLMs without any friction.
https://www.dwarkesh.com/p/mark-zuckerberg#:~:text=As%20long...
The short version is that is you give a product to open source, they can and will donate time and money to improving your product, and the ecosystem around it, for free, and you get to reap those benefits. Llama has already basically won that space (the standard way of running open models is llama.cpp), so OpenAI have finally realized they're playing catch-up (and last quarter's SOTA isn't worth much revenue to them when there's a new SOTA, so they may as well give it away while it can still crack into the market)
Why do companies release open source LLMs? I would understand it, if there was some technology lock-in. But with LLMs, there is no such thing. One can switch out LLMs without any friction.
LLMs are terrible, purely speaking from the business economic side of things. Frontier / SOTA models are barely profitable. Previous gen model lose 90% of their value. Two gens back and they're worthless. And given that their product life cycle is something like 6-12 months, you might as well open source them as part of sundowning them.
Earlier quoted context omitted.
GPT-5 coming Thursday.
How much hype do we anticipate with the release of GPT-5 or whichever name to be included? And how many new features?
Earlier quoted context omitted.
Generation is usually fast, but prompt processing is the main limitation with local agents. I also have a 128 GB M4 Max. How is the prompt processing on long prompts? processing the system prompt for Goose always takes quite a while for me. I haven't been able to download the 120B yet, but I'm looking to switch to either that or the GLM-4.5-Air for my main driver.
it's odd that the result of this processing cannot be cached.
Getting great performance running gpt-oss on 3x A4000's: gpt-oss:20b = ~46 tok/s More than 2x faster than my previous leading OSS models: mistral-small3.2:24b = ~22 tok/s gemma3:27b = ~19.5 tok/s Strangely getting nearly the opposite performance running on 1x 5070 Ti: mistral-small3.2:24b = ~39 tok/s gpt-oss:20b = ~21 tok/s Where gpt-oss is nearly 2x slow vs mistral-small 3.2.
Earlier quoted context omitted.
Okay I will be honest, I was so hyped up about This model but then I went to localllama and saw it that the: 120 B model is worse at coding compared to qwen 3 coder and glm45 air and even grok 3... ( https://www.reddit.com/r/LocalLLaMA/comments/1mig58x/gptoss1... )
That's SVGBench, which is a useful benchmark but isn't much of a test of general coding
Thanks.
guys, what does OSS stand for?