Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

GPT-4.1 in the API

openai.com

Post reply on HN
Print view View original

241–250 of 513 posts

Re: GPT-4.1 in the API

#241

Post by yokto »

cheschire wrote: post #233
999900000999 wrote: post #227

Have they implemented "I don't know" yet. I probably spend 100$ a month on AI coding, and it's great at small straightforward tasks. Drop it into a larger codebase and it'll get confused. Even if the same tool built it in the first place due to context limits. Then again, the way things are rapidly improving I suspect I can wait 6 months and they'll have a model that can do what I want.

I wonder if documentation would help to create an carefully and intentionally tokenized overview of the system. Maximize the amount of routine larger scope information provided in minimal tokens in order to leave room for more immediate context. Similar to the function documentation provides to developers today, I suppose.

It does, shockingly well in my experience. Check out this blog post outlining such an approach, called Literate Development by the author: https://news.ycombinator.com/item?id=43524673

Re: GPT-4.1 in the API

#242

Post by tengbretson »

forbiddenvoid wrote: post #129
taikahessu wrote: post #124

> They feature a refreshed knowledge cutoff of June 2024. As opposed to Gemini 2.5 Pro having cutoff of Jan 2025. Honestly this feels underwhelming and surprising. Especially if you're coding with frameworks with breaking changes, this can hurt you.

It's definitely an issue. Even the simplest use case of "create React app with Vite and Tailwind" is broken with these models right now because they're not up to date.

A few weeks back I couldn't even get ChatGPT to output TypeScript code that correctly used the OpenAI SDK.

Re: GPT-4.1 in the API

#243

Post by rvnx »

oofbaroomf wrote: post #89
poormathskills wrote: post #74

Earlier quoted context omitted.

Go look at their past blog posts. OpenAI only ever benchmarks against their own models.

Oh, ok. But it's still quite telling of their attitude as an organization.

It's the same organization that kept repeating that sharing weights of GPT would be "too dangerous for the world". Eventually DeepSeek thankfully did something like that, though they are supposed to be the evil guys.

Re: GPT-4.1 in the API

#244

Post by behnamoh »

swyx wrote: post #203

don't miss that OAI also published a prompting guide WITH RECEIPTS for GPT 4.1 specifically for those building agents... with a new recommendation for: - telling the model to be persistent (+20%) - dont self-inject/parse toolcalls (+2%) - prompted planning (+4%) - JSON BAD - use XML or arxiv 2406.13121 (GDM format) - put instructions + user query at TOP -and- BOTTOM - bottom-only is VERY BAD - no evidence that ALL CA…

> - JSON BAD - use XML or arxiv 2406.13121 (GDM format)

And yet, all function calling and MCP is done through JSON...

Re: GPT-4.1 in the API

#245

Post by omneity »

kmeisthax wrote: post #117
elashri wrote: post #75

Are there any benchmarks or someone who did tests of performance of using this long max token models in scenarios where you actually use more of this token limit? I found from my experience with Gemini models that after ~200k that the quality drops and that it basically doesn't keep track of things. But I don't have any numbers or systematic study of this behavior. I think all providers who announce increased max tok…

The problem is that while you can train a model with the hyperparameter of "context size" set to 1M, there's very little 1M data to train on. Most of your model's ability to follow long context comes from the fact that it's trained on lots of (stolen) books; in fact I believe OpenAI just outright said in court that they can't do long context without training on books. Novels are usually measured in terms of words; an…

I'm not sure to which extent this opinion is accurately informed. It is well known that nobody trains on 1M token-long content. It wouldn't work anyway as the dependencies are too far fetched and you end up with vanishing gradients.

RoPE (Rotary Positional Embeddings, think modulo or periodic arithmetics) scaling is key, whereby the model is trained on 16k tokens long content, and then scaled up to 100k+ [0]. Qwen 1M (who has near perfect recall over the complete window [1]) and Llama 4 10M pushed the limits of this technique, with Qwen reliably training with a much higher RoPE base, and Llama 4 coming up with iRoPE which claims scaling to extremely long contexts up to infinity.

[0]: https://arxiv.org/html/2310.05209v2

[1]: https://qwenlm.github.io/blog/qwen2.5-turbo/#passkey-retriev...

Re: GPT-4.1 in the API

#246

Post by zoogeny »

pton_xd wrote: post #224
swyx wrote: post #203

don't miss that OAI also published a prompting guide WITH RECEIPTS for GPT 4.1 specifically for those building agents... with a new recommendation for: - telling the model to be persistent (+20%) - dont self-inject/parse toolcalls (+2%) - prompted planning (+4%) - JSON BAD - use XML or arxiv 2406.13121 (GDM format) - put instructions + user query at TOP -and- BOTTOM - bottom-only is VERY BAD - no evidence that ALL CA…

As an aside, one of the worst aspects of the rise of LLMs, for me, has been the wholesale replacement of engineering with trial-and-error hand-waving. Try this, or maybe that, and maybe you'll see a +5% improvement. Why? Who knows. It's just not how I like to work.

I think trial-and-error hand-waving isn't all that far from experimentation.

As an aside, I was working in the games industry when multi-core was brand new. Maybe Xbox-360 and PS3? I'm hazy on the exact consoles but there was one generation where the major platforms all went multi-core.

No one knew how to best use the multi-core systems for gaming. I attended numerous tech talks by teams that had tried different approaches and were give similar "maybe do this and maybe see x% improvement?". There was a lot of experimentation. It took a few years before things settled and best practices became even somewhat standardized.

Some people found that era frustrating and didn't like to work in that way. Others loved the fact it was a wide open field of study where they could discover things.

Re: GPT-4.1 in the API

#247

Post by soheil »

modeless wrote: post #59

Numbers for SWE-bench Verified, Aider Polyglot, cost per million output tokens, output tokens per second, and knowledge cutoff month/year: SWE Aider Cost Fast Fresh Claude 3.7 70% 65% $15 77 8/24 Gemini 2.5 64% 69% $10 200 1/25 GPT-4.1 55% 53% $8 169 6/24 DeepSeek R1 49% 57% $2.2 22 7/24 Grok 3 Beta ? 53% $15 ? 11/24 I'm not sure this is really an apples-to-apples comparison as it may involve different test scaffoldi…

Yes on both Cursor and Windsurf.

https://twitter.com/cursor_ai/status/1911835651810738406

Re: GPT-4.1 in the API

#248

Post by philipbjorge »

pcwelder wrote: post #99

Can someone explain to me why we should take Aider's polyglot benchmark seriously? All the solutions are already available on the internet on which various models are trained, albeit in various ratios. Any variance could likely be due to the mix of the data.

If you're looking to test an LLMs ability to solve a coding task without prior knowledge of the task at hand, I don't think their benchmark is super useful.

If you care about understanding relative performance between models for solving known problems and producing correct output format, it's pretty useful.

- Even for well-known problems, we see a large distribution of quality between models (5 to 75% correctness) - Additionally, we see a large distribution of model's ability to produce responses in formats they were instructed in

At the end of the day, benchmarks are pretty fuzzy, but I always welcome a formalized benchmark as a means to understand model performance over vibe checking.

Re: GPT-4.1 in the API

#250

Post by kgeist »

growt wrote: post #210

My theory: they need to move off the 4o version number before releasing o4-mini next week or so.

The 'oN' schema was a such strange choice for branding. They had to skip 'o2' because it's already trademarked, and now 'o4' can easily be confused with '4o'.
Post reply on HN