71–78 of 78 posts
On flights with shitty wifi I have been running gpt-oss:120b on my macbook using ollama. Ok model for coding if you can't reach a good one.
Are you running the full 65GB model on a MacBook Pro? What tokens per second do you get? What specs? M5?
Claude has had an uncomfortable number of availability incidents recently. https://status.claude.com/
Similarly published by OpenAI: https://status.openai.com/
30 day comparisons as of writing:
99.61% for Claude.ai 99.22% for ChatGPT
99.92% for Claude APIs 99.25% for OpenAI APIs
Obviously not apples to apples and somewhat up to discretion of what triggers an impact. We’re clearly not at 99.99% yet.
Earlier quoted context omitted.
Wow, thanks for the info. I'm planning on testing this on my M4 Max w/ 36 GB today. edit: So looking here https://ollama.com/library/gpt-oss/tags it seems ollama doesn't even provide the MXFP4 variants, much less hide them. Is the best way to run these variants via llama.cpp or...?
on the model description page they claim they support it: Quantization - MXFP4 format OpenAI utilizes quantization to reduce the memory footprint of the gpt-oss models. The models are post-trained with quantization of the mixture-of-experts (MoE) weights to MXFP4 format, where the weights are quantized to 4.25 bits per parameter. The MoE weights are responsible for 90+% of the total parameter count, and quantizing th…
Earlier quoted context omitted.
on the model description page they claim they support it: Quantization - MXFP4 format OpenAI utilizes quantization to reduce the memory footprint of the gpt-oss models. The models are post-trained with quantization of the mixture-of-experts (MoE) weights to MXFP4 format, where the weights are quantized to 4.25 bits per parameter. The MoE weights are responsible for 90+% of the total parameter count, and quantizing th…
Can you link to that page? I’m not finding these variants.
On flights with shitty wifi I have been running gpt-oss:120b on my macbook using ollama. Ok model for coding if you can't reach a good one.
GPT-OSS-120b/20b is probably the best you can run on your own hardware today. Be careful with the quantized versions though, as they're really horrible compared to the native MXFP4. I haven't looked in this particular case, but Ollama tends to hide their quantizations for some reason, so most people who could be running 20B with MXFP4, are still on Q8 and getting much worse results than they could.
Most gpt-oss GGUF files online have parts of their weights quantized to q8_0, and we've seen folks get some strange results from these models. If you're importing these to Ollama to run, the output quality may decrease.
Earlier quoted context omitted.
It's a different way of doing quantization ( https://huggingface.co/docs/transformers/en/quantization/mxf... ) but I think the most important thing is that OpenAI delivered their own quantization (the MXFP4 from OpenAI/GPT-OSS on HuggingFace, guaranteed correct) whereas all the Q8 and other quantizations you see floating around are community efforts, with somewhat uneven results depending on who done it. Concretely f…
Wow, thanks for the info. I'm planning on testing this on my M4 Max w/ 36 GB today. edit: So looking here https://ollama.com/library/gpt-oss/tags it seems ollama doesn't even provide the MXFP4 variants, much less hide them. Is the best way to run these variants via llama.cpp or...?
You can use the command `ollama show -v gpt-oss:120b` to see the datatype of each tensor.
Earlier quoted context omitted.
Could you share which Macbook model? And what context size you're getting.
I just checked gpt-oss:20b on my M4 Pro 24GB, and got 400.67 tokens/s on input and 46.53 tokens/s on output. That's for a tiny context of 72 tokens.
Earlier quoted context omitted.
> all the AI API's statii The Latin plural of "status", in the accusative form, would actually be "status" as well. Something like omnes status intelligentiae artificialis in eadem pagina videre amem.
Life of Brian :) https://youtu.be/DdqXT9k-050?si=L5ymXl-fYe7Fjqye