161–170 of 513 posts
Numbers for SWE-bench Verified, Aider Polyglot, cost per million output tokens, output tokens per second, and knowledge cutoff month/year: SWE Aider Cost Fast Fresh Claude 3.7 70% 65% $15 77 8/24 Gemini 2.5 64% 69% $10 200 1/25 GPT-4.1 55% 53% $8 169 6/24 DeepSeek R1 49% 57% $2.2 22 7/24 Grok 3 Beta ? 53% $15 ? 11/24 I'm not sure this is really an apples-to-apples comparison as it may involve different test scaffoldi…
No benchmark comparisons to other models, especially Gemini 2.5 Pro, is telling.
Numbers for SWE-bench Verified, Aider Polyglot, cost per million output tokens, output tokens per second, and knowledge cutoff month/year: SWE Aider Cost Fast Fresh Claude 3.7 70% 65% $15 77 8/24 Gemini 2.5 64% 69% $10 200 1/25 GPT-4.1 55% 53% $8 169 6/24 DeepSeek R1 49% 57% $2.2 22 7/24 Grok 3 Beta ? 53% $15 ? 11/24 I'm not sure this is really an apples-to-apples comparison as it may involve different test scaffoldi…
Looks like they also added the cost of the benchmark run to the leaderboard, which is quite cool. Cost per output token is no longer representative of the actual cost when the number of tokens can vary by an order of magnitude for the same problem just based on how many thinking tokens the model is told to use.
Earlier quoted context omitted.
Btw, as someone who agrees with your point, what’s the actual answer to this?
Of these, some are mostly obsolete: GPT-4 and GPT-4 Turbo are worse than GPT-4o in both speed and capabilities. o1 is worse than o3-mini-high in most aspects. Then, some are not available yet: o3 and o4-mini. GPT-4.1 I haven't played with enough to give you my opinion on. Among the rest, it depends on what you're looking for: Multi-modal: GPT-4o > everything else Reasoning: o1-pro > o3-mini-high > o3-mini Speed: GPT-…
> GPT‑4.5 Preview will be turned off in three months, on July 14, 2025
With these being 1M context size, does that all but confirm that Quasar Alpha and Optimus Alpha were cloaked OpenAI models on OpenRouter?
Lots of improvements here (hopefully), but still no image generation updates, which is what I'm most eager for right now.
Earlier quoted context omitted.
I have a medium-sized typescript personal project I work on. It probably has 20k LOC of well organized typescript (react frontend, express backend). I also have somewhat comprehensive docs and cursor project rules. In general I use Cursor in manual mode asking it to make very well scoped small changes (e.g. “write this function that does this in this exact spot”). Yesterday I needed to make a largely mechanical chang…
My personal opinion is leveraging LLMs on a large code base requires skill. How you construct the prompt, and what you keep in context, which model you use, all have a large effect on the output. If you just put it into cursor and throw your hands up, you probably didnt do it right
Earlier quoted context omitted.
Gemini 2.5 Pro gets 64% on SWE-bench verified. Sonnet 3.7 gets 70% They are reporting that GPT-4.1 gets 55%.
Are those with «thinking» or without?