Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

GPT-4.1 in the API

openai.com

Post reply on HN
Print view View original

181–190 of 513 posts

Re: GPT-4.1 in the API

#181

Post by kevmo314 »

marsh_mellow wrote: post #174
InkCanon wrote: post #149

Earlier quoted context omitted.

>4.1 Was better in 55% of cases Um, isn't that just a fancy way of saying it is slightly better >Score of 6.81 against 6.66 So very slightly better

I don't think the absolute score means much — judge models have a tendency to score around 7/10 lol 55% vs. 45% equates to about a 36 point difference in ELO. in chess that would be two players in the same league but one with a clear edge

Rarely are two models put head-to-head though. If Claude Sonnet 3.7 isn't able to generate a good PR review (for whatever reason), a 2% better review isn't all that strong of a value proposition.

Re: GPT-4.1 in the API

#182

Post by simonw »

Here's a summary of this Hacker News thread created by GPT-4.1 (the full sized model) when the conversation hit 164 comments: https://gist.github.com/simonw/93b2a67a54667ac46a247e7c5a2fe...

I think it did very well - it's clearly good at instruction following.

Total token cost: 11,758 input, 2,743 output = 4.546 cents.

Same experiment run with GPT-4.1 mini: https://gist.github.com/simonw/325e6e5e63d449cc5394e92b8f2a3... (0.8802 cents)

And GPT-4.1 nano: https://gist.github.com/simonw/1d19f034edf285a788245b7b08734... (0.2018 cents)

Re: GPT-4.1 in the API

#183

Post by wiz21c »

InkCanon wrote: post #149
marsh_mellow wrote: post #142

From OpenAI's announcement: > Qodo tested GPT‑4.1 head-to-head against Claude Sonnet 3.7 on generating high-quality code reviews from GitHub pull requests. Across 200 real-world pull requests with the same prompts and conditions, they found that GPT‑4.1 produced the better suggestion in 55% of cases. Notably, they found that GPT‑4.1 excels at both precision (knowing when not to make suggestions) and comprehensiveness…

>4.1 Was better in 55% of cases Um, isn't that just a fancy way of saying it is slightly better >Score of 6.81 against 6.66 So very slightly better

"they found that GPT‑4.1 excels at both precision..."

They didn't say it is better than Claude at precision etc. Just that it excels.

Unfortunately, AI has still not concluded that manipulations by the marketing dept is a plague...

Re: GPT-4.1 in the API

#184

Post by TIPSIO »

taikahessu wrote: post #124

> They feature a refreshed knowledge cutoff of June 2024. As opposed to Gemini 2.5 Pro having cutoff of Jan 2025. Honestly this feels underwhelming and surprising. Especially if you're coding with frameworks with breaking changes, this can hurt you.

It it annoying. The bigger cheaper context windows help this a little though:

E.g.: If context windows get big and cheap enough (as things are trending), hopefully you can just dump the entire docs, examples, and more in every request.

Re: GPT-4.1 in the API

#185

Post by modeless »

jsnell wrote: post #163
modeless wrote: post #59

Numbers for SWE-bench Verified, Aider Polyglot, cost per million output tokens, output tokens per second, and knowledge cutoff month/year: SWE Aider Cost Fast Fresh Claude 3.7 70% 65% $15 77 8/24 Gemini 2.5 64% 69% $10 200 1/25 GPT-4.1 55% 53% $8 169 6/24 DeepSeek R1 49% 57% $2.2 22 7/24 Grok 3 Beta ? 53% $15 ? 11/24 I'm not sure this is really an apples-to-apples comparison as it may involve different test scaffoldi…

https://aider.chat/docs/leaderboards/ shows 73% rather than 69% for Gemini 2.5 Pro? Looks like they also added the cost of the benchmark run to the leaderboard, which is quite cool. Cost per output token is no longer representative of the actual cost when the number of tokens can vary by an order of magnitude for the same problem just based on how many thinking tokens the model is told to use.

There are different scores reported by Google for "diff" and "whole" modes, and the others were "diff" so I chose the "diff" score. Hard to make a real apples-to-apples comparison.

Re: GPT-4.1 in the API

#186

Post by rob »

simianwords wrote: post #140
asdev wrote: post #108

> We will also begin deprecating GPT‑4.5 Preview in the API, as GPT‑4.1 offers improved or similar performance on many key capabilities at much lower cost and latency. why would they deprecate when it's the better model? too expensive?

Where did you find that 4.5 is a better model? Everything from the video told me that 4.5 was largely a mistake and 4.1 beats 4.5 at everything. There's no point keeping 4.5 at this point.

Bigger numbers are supposed to mean better. 3.5, 4, 4.5. Going from 4 to 4.5 to 4.1 seems weird to most people. If it's better, it should of been GPT-4.6 or 5.0 or something else, not a downgraded number.

Re: GPT-4.1 in the API

#187

Post by jsnell »

marsh_mellow wrote: post #142

From OpenAI's announcement: > Qodo tested GPT‑4.1 head-to-head against Claude Sonnet 3.7 on generating high-quality code reviews from GitHub pull requests. Across 200 real-world pull requests with the same prompts and conditions, they found that GPT‑4.1 produced the better suggestion in 55% of cases. Notably, they found that GPT‑4.1 excels at both precision (knowing when not to make suggestions) and comprehensiveness…

That's not a lot of samples for such a small effect, I don't think it's statistically significant (p-value of around 10%).

Re: GPT-4.1 in the API

#188

Post by usaar333 »

asdev wrote: post #67

it's worse than 4.5 on nearly every benchmark. just an incremental improvement. AI is slowing down

Or OpenAI is? After using Gemini 2.5, I did not feel "AI is slowing down". It's just this model isn't SOTA.

Re: GPT-4.1 in the API

#189

Post by atemerev »

Tinkeringz wrote: post #173
atemerev wrote: post #90

Earlier quoted context omitted.

Yes, confirmed by citing Aider benchmarks: https://openai.com/index/gpt-4-1/ Which means that these models are _absolutely_ not SOTA, and Gemini 2.5 pro is much better, and Sonnet is better, and even R1 is better. Sorry Sam, you are losing the game.

Aren’t all of these reasoning models? Won’t the reasoning models of openAI benchmarked against these be a test of if Sam is losing?

There is no OpenAI model better than R1, reasoning or not (as confirmed by the same Aider benchmark; non-coding tests are less objective, but I think it still holds).

With Gemini (current SOTA) and Sonnet (great potential, but tends to overengineer/overdo things) it is debatable, they are probably better than R1 (and all OpenAI models by extension).

Re: GPT-4.1 in the API

#190

Post by Zambyte »

forbiddenvoid wrote: post #129
taikahessu wrote: post #124

> They feature a refreshed knowledge cutoff of June 2024. As opposed to Gemini 2.5 Pro having cutoff of Jan 2025. Honestly this feels underwhelming and surprising. Especially if you're coding with frameworks with breaking changes, this can hurt you.

It's definitely an issue. Even the simplest use case of "create React app with Vite and Tailwind" is broken with these models right now because they're not up to date.

By "broken" you mean it doesn't use the latest and greatest hot trend, right? Or does it literally not work?
Post reply on HN