Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

GPT-4.1 in the API

openai.com

Post reply on HN
Print view View original

221–230 of 513 posts

Re: GPT-4.1 in the API

#221

Post by modeless »

jsnell wrote: post #217
modeless wrote: post #197

Earlier quoted context omitted.

Huh, seems like Aider made a special mode specifically for Gemini[1] some time after Google's announcement blog post with official performance numbers. Still not sure it makes sense to quote that new score next to the others. In any case Gemini's 69% is the top score even without a special mode. [1] https://aider.chat/docs/more/edit-formats.html#diff-fenced:~...

The mode wasn't added after the announcement, Aider has had it for almost a year: https://aider.chat/HISTORY.html#aider-v0320 This benchmark has an authoritative source of results (the leaderboard), so it seems obvious that it's the number that should be used.

OK but it was still added specifically to improve Gemini and nobody else on the leaderboard uses it. Google themselves do not use it when they benchmark their own models against others. They use the regular diff mode that everyone else uses. https://blog.google/technology/google-deepmind/gemini-model-...

Re: GPT-4.1 in the API

#222

Post by kweingar »

poormathskills wrote: post #70
ZeroCool2u wrote: post #8

No benchmark comparisons to other models, especially Gemini 2.5 Pro, is telling.

Go look at their past blog posts. OpenAI only ever benchmarks against their own models. This is pretty common across industries. The leader doesn’t compare themselves to the competition.

That would make sense if OAI were the leader.

Re: GPT-4.1 in the API

#224

Post by pton_xd »

swyx wrote: post #203

don't miss that OAI also published a prompting guide WITH RECEIPTS for GPT 4.1 specifically for those building agents... with a new recommendation for: - telling the model to be persistent (+20%) - dont self-inject/parse toolcalls (+2%) - prompted planning (+4%) - JSON BAD - use XML or arxiv 2406.13121 (GDM format) - put instructions + user query at TOP -and- BOTTOM - bottom-only is VERY BAD - no evidence that ALL CA…

As an aside, one of the worst aspects of the rise of LLMs, for me, has been the wholesale replacement of engineering with trial-and-error hand-waving. Try this, or maybe that, and maybe you'll see a +5% improvement. Why? Who knows.

It's just not how I like to work.

Re: GPT-4.1 in the API

#225

Post by marsh_mellow »

jsnell wrote: post #187
marsh_mellow wrote: post #142

From OpenAI's announcement: > Qodo tested GPT‑4.1 head-to-head against Claude Sonnet 3.7 on generating high-quality code reviews from GitHub pull requests. Across 200 real-world pull requests with the same prompts and conditions, they found that GPT‑4.1 produced the better suggestion in 55% of cases. Notably, they found that GPT‑4.1 excels at both precision (knowing when not to make suggestions) and comprehensiveness…

That's not a lot of samples for such a small effect, I don't think it's statistically significant (p-value of around 10%).

p-value of 7.9% — so very close to statistical significance.

the p-value for GPT-4.1 having a win rate of at least 49% is 4.92%, so we can say conclusively that GPT-4.1 is at least (essentially) evenly matched with Claude Sonnet 3.7, if not better.

Given that Claude Sonnet 3.7 has been generally considered to be the best (non-reasoning) model for coding, and given that GPT-4.1 is substantially cheaper ($2/million input, $8/million output vs. $3/million input, $15/million output), I think it's safe to say that this is significant news, although not a game changer

Re: GPT-4.1 in the API

#226

Post by swyx »

simonw wrote: post #211
swyx wrote: post #203

don't miss that OAI also published a prompting guide WITH RECEIPTS for GPT 4.1 specifically for those building agents... with a new recommendation for: - telling the model to be persistent (+20%) - dont self-inject/parse toolcalls (+2%) - prompted planning (+4%) - JSON BAD - use XML or arxiv 2406.13121 (GDM format) - put instructions + user query at TOP -and- BOTTOM - bottom-only is VERY BAD - no evidence that ALL CA…

I'm surprised and a little disappointed by the result concerning instructions at the top, because it's incompatible with prompt caching: I would much rather cache the part of the prompt that includes the long document and then swap out the user question at the end.

yep. we address it in the podcast. presumably this is just a recent discovery and can be post-trained away.

Re: GPT-4.1 in the API

#227

Post by 999900000999 »

Have they implemented "I don't know" yet.

I probably spend 100$ a month on AI coding, and it's great at small straightforward tasks.

Drop it into a larger codebase and it'll get confused. Even if the same tool built it in the first place due to context limits.

Then again, the way things are rapidly improving I suspect I can wait 6 months and they'll have a model that can do what I want.

Re: GPT-4.1 in the API

#228

Post by zaptrem »

simonw wrote: post #211
swyx wrote: post #203

don't miss that OAI also published a prompting guide WITH RECEIPTS for GPT 4.1 specifically for those building agents... with a new recommendation for: - telling the model to be persistent (+20%) - dont self-inject/parse toolcalls (+2%) - prompted planning (+4%) - JSON BAD - use XML or arxiv 2406.13121 (GDM format) - put instructions + user query at TOP -and- BOTTOM - bottom-only is VERY BAD - no evidence that ALL CA…

I'm surprised and a little disappointed by the result concerning instructions at the top, because it's incompatible with prompt caching: I would much rather cache the part of the prompt that includes the long document and then swap out the user question at the end.

Prompt on bottom is also easier for humans to read as I can have my actual question and the model’s answer on screen at the same time instead of scrolling through 70k tokens of context between them.

Re: GPT-4.1 in the API

#229

Post by HDThoreaun »

rob wrote: post #186
simianwords wrote: post #140

Earlier quoted context omitted.

Where did you find that 4.5 is a better model? Everything from the video told me that 4.5 was largely a mistake and 4.1 beats 4.5 at everything. There's no point keeping 4.5 at this point.

Bigger numbers are supposed to mean better. 3.5, 4, 4.5. Going from 4 to 4.5 to 4.1 seems weird to most people. If it's better, it should of been GPT-4.6 or 5.0 or something else, not a downgraded number.

OpenAI has decided to troll via crappy naming conventions as a sort of in joke. Sam Altman tweets about it pretty often

Re: GPT-4.1 in the API

#230

Post by OxfordOutlander »

nikcub wrote: post #166

Easy to miss in the announcement that 4.5 is being shut down > GPT‑4.5 Preview will be turned off in three months, on July 14, 2025

Juice not worth the squeeze I imagine. 4.5 is chonky, and having to reserve GPU space for it must not have been worth it. Makes sense to me - I hadn't founding anything it was so much better at that it was worth the incremental cost over Sonnet 3.7 or o3-mini.
Post reply on HN