411–420 of 513 posts
As a ChatGPT user, I'm weirdly happy that it's not available there yet. I already have to make a conscious choice between - 4o (can search the web, use Canvas, evaluate Python server-side, generate images, but has no chain of thought) - o3-mini (web search, CoT, canvas, but no image generation) - o1 (CoT, maybe better than o3, but no canvas or web search and also no images) - Deep Research (very powerful, but I have…
Are there any benchmarks or someone who did tests of performance of using this long max token models in scenarios where you actually use more of this token limit? I found from my experience with Gemini models that after ~200k that the quality drops and that it basically doesn't keep track of things. But I don't have any numbers or systematic study of this behavior. I think all providers who announce increased max tok…
> For instance, the NoLiMa benchmark revealed that models like GPT-4o experienced a significant drop from a 99.3% performance rate at 1,000 tokens to 69.7% at 32,000 tokens. Similarly, Llama 3.3 70B's effectiveness decreased from 97.3% at 1,000 tokens to 42.7% at 32,000 tokens, highlighting the challenges LLMs face with longer contexts.
Earlier quoted context omitted.
I use them as follows: o1-pro: anything important involving accuracy or reasoning. Does the best at accomplishing things correctly in one go even with lots of context. deepseek R1: anything where I want high quality non-academic prose or poetry. Hands down the best model for these. Also very solid for fast and interesting analytical takes. I love bouncing ideas around with R1 and Grok-3 bc of their fast responses and…
You probably know this but it can already generate accurate diagrams. Just ask for the output in a diagram language like mermaid or graphviz
Are there any benchmarks or someone who did tests of performance of using this long max token models in scenarios where you actually use more of this token limit? I found from my experience with Gemini models that after ~200k that the quality drops and that it basically doesn't keep track of things. But I don't have any numbers or systematic study of this behavior. I think all providers who announce increased max tok…
Is there a reliable method for pruning, summarizing, or otherwise compressing context to overcome such issues?
Earlier quoted context omitted.
Okay, it's common across other industries, but not this one. Here is Google, Facebook, and Anthropic comparing their frontier models to others[1][2][3]. [1] https://blog.google/technology/google-deepmind/gemini-model-... [2] https://ai.meta.com/blog/llama-4-multimodal-intelligence/ [3] https://www.anthropic.com/claude/sonnet
Right. Those labs aren’t leading the industry.
Have they implemented "I don't know" yet. I probably spend 100$ a month on AI coding, and it's great at small straightforward tasks. Drop it into a larger codebase and it'll get confused. Even if the same tool built it in the first place due to context limits. Then again, the way things are rapidly improving I suspect I can wait 6 months and they'll have a model that can do what I want.
I believe this. I've been having the forgetting problem happen less with Gemini 2.5 Pro. It does hallucinate, but I can get far just pasting all the docs and a few examples, and asking it to double check everything according to the docs instead of relying on its memory.
As a ChatGPT user, I'm weirdly happy that it's not available there yet. I already have to make a conscious choice between - 4o (can search the web, use Canvas, evaluate Python server-side, generate images, but has no chain of thought) - o3-mini (web search, CoT, canvas, but no image generation) - o1 (CoT, maybe better than o3, but no canvas or web search and also no images) - Deep Research (very powerful, but I have…
Earlier quoted context omitted.
Yes, confirmed by citing Aider benchmarks: https://openai.com/index/gpt-4-1/ Which means that these models are _absolutely_ not SOTA, and Gemini 2.5 pro is much better, and Sonnet is better, and even R1 is better. Sorry Sam, you are losing the game.
Aren’t all of these reasoning models? Won’t the reasoning models of openAI benchmarked against these be a test of if Sam is losing?
As a ChatGPT user, I'm weirdly happy that it's not available there yet. I already have to make a conscious choice between - 4o (can search the web, use Canvas, evaluate Python server-side, generate images, but has no chain of thought) - o3-mini (web search, CoT, canvas, but no image generation) - o1 (CoT, maybe better than o3, but no canvas or web search and also no images) - Deep Research (very powerful, but I have…