381–390 of 513 posts
- It's basically GPT4o level on average.
- More optimized for coding, but slightly inferior in other areas.
It seems to be a better model than 4o for coding tasks, but I'm not sure if it will replace the current leaders -- Gemini 2.5 Pro, o3-mini / o1, Claude 3.7/3.5.
Earlier quoted context omitted.
Time to start moving back to Java & Spring. 100% backwards compatibility and well represented in 15 years worth of training data, hah.
Write once, run nowhere.
(I did use Spring, once, ages ago, and we deployed the app to a local Tomcat server in the office...)
Earlier quoted context omitted.
Gemini 2.5 Pro gets 64% on SWE-bench verified. Sonnet 3.7 gets 70% They are reporting that GPT-4.1 gets 55%.
Are those with «thinking» or without?
don't miss that OAI also published a prompting guide WITH RECEIPTS for GPT 4.1 specifically for those building agents... with a new recommendation for: - telling the model to be persistent (+20%) - dont self-inject/parse toolcalls (+2%) - prompted planning (+4%) - JSON BAD - use XML or arxiv 2406.13121 (GDM format) - put instructions + user query at TOP -and- BOTTOM - bottom-only is VERY BAD - no evidence that ALL CA…
I'm surprised and a little disappointed by the result concerning instructions at the top, because it's incompatible with prompt caching: I would much rather cache the part of the prompt that includes the long document and then swap out the user question at the end.
If the instructions are at the top the LV cache entries can be pre computed and cached.
If they’re at the bottom the entries at the lower layers will have a dependency on the user input.
Numbers for SWE-bench Verified, Aider Polyglot, cost per million output tokens, output tokens per second, and knowledge cutoff month/year: SWE Aider Cost Fast Fresh Claude 3.7 70% 65% $15 77 8/24 Gemini 2.5 64% 69% $10 200 1/25 GPT-4.1 55% 53% $8 169 6/24 DeepSeek R1 49% 57% $2.2 22 7/24 Grok 3 Beta ? 53% $15 ? 11/24 I'm not sure this is really an apples-to-apples comparison as it may involve different test scaffoldi…
Deepseek for general chat and research Claude 3.7 for coding Gemini 2.5 Pro experimental for deep research
In terms of price Deepseek is still absolutely fire!
OpenAI is in trouble honestly.
Here's a summary of this Hacker News thread created by GPT-4.1 (the full sized model) when the conversation hit 164 comments: https://gist.github.com/simonw/93b2a67a54667ac46a247e7c5a2fe... I think it did very well - it's clearly good at instruction following. Total token cost: 11,758 input, 2,743 output = 4.546 cents. Same experiment run with GPT-4.1 mini: https://gist.github.com/simonw/325e6e5e63d449cc5394e92b8f2a3…
As a ChatGPT user, I'm weirdly happy that it's not available there yet. I already have to make a conscious choice between - 4o (can search the web, use Canvas, evaluate Python server-side, generate images, but has no chain of thought) - o3-mini (web search, CoT, canvas, but no image generation) - o1 (CoT, maybe better than o3, but no canvas or web search and also no images) - Deep Research (very powerful, but I have…
Earlier quoted context omitted.
> - Deep Research (very powerful, but I have only 10 attempts per month, so I end up using roughly zero) Same here, which is a real shame. I've switched to DeepResearch with Gemini 2.5 Pro over the last few days where paid users have a 20/day limit instead of 10/month and it's been great, especially since now Gemini seems to browse 10x more pages than OpenAI Deep Research (on the order of 200-400 pages versus 20-40).…
I also like Perplexity’s 3/day limit! If I use them up (which I almost never do) I can just refresh the next day
Earlier quoted context omitted.
I use them as follows: o1-pro: anything important involving accuracy or reasoning. Does the best at accomplishing things correctly in one go even with lots of context. deepseek R1: anything where I want high quality non-academic prose or poetry. Hands down the best model for these. Also very solid for fast and interesting analytical takes. I love bouncing ideas around with R1 and Grok-3 bc of their fast responses and…
Gemini 2.5 Pro is quite good at code. Has become my go to for use in Cursor. Claude 3.7 needs to be restrained too much.