Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

GPT-4.1 in the API

openai.com

Post reply on HN
Print view View original

191–200 of 513 posts

Re: GPT-4.1 in the API

#191

Post by crimsoneer »

kmeisthax wrote: post #117
elashri wrote: post #75

Are there any benchmarks or someone who did tests of performance of using this long max token models in scenarios where you actually use more of this token limit? I found from my experience with Gemini models that after ~200k that the quality drops and that it basically doesn't keep track of things. But I don't have any numbers or systematic study of this behavior. I think all providers who announce increased max tok…

The problem is that while you can train a model with the hyperparameter of "context size" set to 1M, there's very little 1M data to train on. Most of your model's ability to follow long context comes from the fact that it's trained on lots of (stolen) books; in fact I believe OpenAI just outright said in court that they can't do long context without training on books. Novels are usually measured in terms of words; an…

Isn't the problem more that the "needle in a haystack" eval (i said word X once, where) is really not relevant to most long context LLM use cases like code, where you need the context from all the stuff simultaneously rather than identifying a single, quite separate relevant section?

Re: GPT-4.1 in the API

#192

Post by jsnell »

modeless wrote: post #185
jsnell wrote: post #163

Earlier quoted context omitted.

https://aider.chat/docs/leaderboards/ shows 73% rather than 69% for Gemini 2.5 Pro? Looks like they also added the cost of the benchmark run to the leaderboard, which is quite cool. Cost per output token is no longer representative of the actual cost when the number of tokens can vary by an order of magnitude for the same problem just based on how many thinking tokens the model is told to use.

There are different scores reported by Google for "diff" and "whole" modes, and the others were "diff" so I chose the "diff" score. Hard to make a real apples-to-apples comparison.

The 73% on the current leaderboard is using "diff", not "whole". (Well, diff-fenced, but the difference is just the location of the filename.)

Re: GPT-4.1 in the API

#193

Post by croemer »

gs17 wrote: post #102
croemer wrote: post #33

Testing against unspecified other "leading" models allows for shenanigangs: > Qodo tested GPT‑4.1 head-to-head against other leading models [...] they found that GPT‑4.1 produced the better suggestion in 55% of cases The linked blog post goes 404: https://www.qodo.ai/blog/benchmarked-gpt-4-1/

The post seems to be up now and seems to compare it slightly favorable to Claude 3.7.

Right, now it's up and comparison against Claude 3.7 is better than I feared based on the wording. Though why does the OpenAI announcement talk of comparison against multiple leading models when the Qodo blog post only tests against Claude 3.7...

Re: GPT-4.1 in the API

#194

Post by henlobenlo »

School-Cotton wrote: post #80
henlobenlo wrote: post #61

Earlier quoted context omitted.

Whats the problem, for the layman it doesnt actually matter, and for the experts, its usually very obvious which model to use.

That’s not true. I’m a layman and 4.5 is obviously better than 4o for me, definitely enough to matter.

You are definitely not a layman if you know the difference between 4.5 and 4o. The average user thinks ai = openai = chatgpt.

Re: GPT-4.1 in the API

#196

Post by pzo »

croemer wrote: post #48
Tiberium wrote: post #42

Very important note: >Note that GPT‑4.1 will only be available via the API. In ChatGPT, many of the improvements in instruction following, coding, and intelligence have been gradually incorporated into the latest version If anyone here doesn't know, OpenAI does offer the ChatGPT model version in the API as chatgpt-4o-latest, but it's bad because they continuously update it so businesses can't reliably rely on it bein…

So you're saying that "ChatGPT-4o-latest (2025-03-26)" in LMarena is 4.1?

yeah I was surprised in they benchmarks during livestream they didn't compare to ChatGPT-4o (2025-03-26) but only older one.

Re: GPT-4.1 in the API

#197

Post by modeless »

jsnell wrote: post #192
modeless wrote: post #185

Earlier quoted context omitted.

There are different scores reported by Google for "diff" and "whole" modes, and the others were "diff" so I chose the "diff" score. Hard to make a real apples-to-apples comparison.

The 73% on the current leaderboard is using "diff", not "whole". (Well, diff-fenced, but the difference is just the location of the filename.)

Huh, seems like Aider made a special mode specifically for Gemini[1] some time after Google's announcement blog post with official performance numbers. Still not sure it makes sense to quote that new score next to the others. In any case Gemini's 69% is the top score even without a special mode.

[1] https://aider.chat/docs/more/edit-formats.html#diff-fenced:~...

Re: GPT-4.1 in the API

#199

Post by nanook »

Tinkeringz wrote: post #168
forbiddenvoid wrote: post #134

Lots of improvements here (hopefully), but still no image generation updates, which is what I'm most eager for right now.

They just realised a new image generation a couple of weeks ago, why are you eager for another one so soon?

Are the image generation improvements available via API? Don't think so

Re: GPT-4.1 in the API

#200

Post by tcdent »

modeless wrote: post #185
jsnell wrote: post #163

Earlier quoted context omitted.

https://aider.chat/docs/leaderboards/ shows 73% rather than 69% for Gemini 2.5 Pro? Looks like they also added the cost of the benchmark run to the leaderboard, which is quite cool. Cost per output token is no longer representative of the actual cost when the number of tokens can vary by an order of magnitude for the same problem just based on how many thinking tokens the model is told to use.

There are different scores reported by Google for "diff" and "whole" modes, and the others were "diff" so I chose the "diff" score. Hard to make a real apples-to-apples comparison.

They just pick the best performer out of the built-in modes they offer.

Interesting data point about the models behavior, but even moreso it's a recommendation of which way to configure the model for optimal performance.

I do consider this to be an apple-to-apples benchmark since they're evaluating real world performance.

Post reply on HN