Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

GPT-4.1 in the API

openai.com

Post reply on HN
Print view View original

391–400 of 513 posts

Re: GPT-4.1 in the API

#391

Post by behnamoh »

bangaladore wrote: post #329
motoboi wrote: post #327

Earlier quoted context omitted.

You probably know this but it can already generate accurate diagrams. Just ask for the output in a diagram language like mermaid or graphviz

My experience is it often produces terrible diagrams. Things clearly overlap, lines make no sense. I'm not surprised as if you told me to layout a diagram in XML/YAML there would be obvious mistakes and layout issues. I'm not really certain a text output model can ever do well here.

I had a latex tikz diagram problem which sonnet 3.7 couldn't handle even after 10 attempts. Gemini 2.5 Pro solved it on the second try.

Re: GPT-4.1 in the API

#392

Post by hackernewds »

zoogeny wrote: post #246
pton_xd wrote: post #224

Earlier quoted context omitted.

As an aside, one of the worst aspects of the rise of LLMs, for me, has been the wholesale replacement of engineering with trial-and-error hand-waving. Try this, or maybe that, and maybe you'll see a +5% improvement. Why? Who knows. It's just not how I like to work.

I think trial-and-error hand-waving isn't all that far from experimentation. As an aside, I was working in the games industry when multi-core was brand new. Maybe Xbox-360 and PS3? I'm hazy on the exact consoles but there was one generation where the major platforms all went multi-core. No one knew how to best use the multi-core systems for gaming. I attended numerous tech talks by teams that had tried different appr…

there probably was still a structured way to test this through cross hatching but yeah like blind guessing might take longer and arrive at the same solution

Re: GPT-4.1 in the API

#393

Post by clbrmbr »

The deprecation of GPT-4.5 makes me sad. It's an amazing model with great world-knowledge and subtly. It KNOWS THINGS that, on a quick experiment, 4.1 just does not. 4.5 could tell me what I would see from a random street corner in New Jersey, or how to use minor features of my niche API (well, almost), and it could write remarkably. But 4.1 doesn't hold a candle to it. Please, continue to charge me $150/1M tokens. Sometimes you need a Big Model. Tells me it was costing more than $150/1M to serve (!).

Re: GPT-4.1 in the API

#394

Post by int_19h »

ezyang wrote: post #367
int_19h wrote: post #271

Earlier quoted context omitted.

As usual with LLMs. In my experience, all those metrics are useful mainly to tell which models are definitely bad, but doesn't tell you much about which ones are good, and especially not how the good ones stack against each other in real world use cases. Andrej Karpathy famously quipped that he only trusts two LLM evals: Chatbot Arena (which has humans blindly compare and score responses), and the r/LocalLLaMA commen…

Lmarena isn't that useful anymore lol

I actually agree with that, but it's generally better than other scores. Also, the quote is like a year old at this point.

In practice you have to evaluate the models yourself for any non-trivial task.

Re: GPT-4.1 in the API

#395

Post by purplerabbit »

anotherpaulg wrote: post #370
modeless wrote: post #59

Numbers for SWE-bench Verified, Aider Polyglot, cost per million output tokens, output tokens per second, and knowledge cutoff month/year: SWE Aider Cost Fast Fresh Claude 3.7 70% 65% $15 77 8/24 Gemini 2.5 64% 69% $10 200 1/25 GPT-4.1 55% 53% $8 169 6/24 DeepSeek R1 49% 57% $2.2 22 7/24 Grok 3 Beta ? 53% $15 ? 11/24 I'm not sure this is really an apples-to-apples comparison as it may involve different test scaffoldi…

I just finished updating the aider polyglot leaderboard [0] with GPT-4.1, mini and nano. My results basically agree with OpenAI's published numbers. Results, with other models for comparison: Model Score Cost Gemini 2.5 Pro Preview 03-25 72.9% $ 6.32 claude-3-7-sonnet-20250219 64.9% $36.83 o3-mini (high) 60.4% $18.16 Grok 3 Beta 53.3% $11.03 * gpt-4.1 52.4% $ 9.86 Grok 3 Mini Beta (high) 49.3% $ 0.73 * gpt-4.1-mini 3…

What model are you personally using in your aider coding? :)

Re: GPT-4.1 in the API

#396

Post by cr4zy »

lxgr wrote: post #317

As a ChatGPT user, I'm weirdly happy that it's not available there yet. I already have to make a conscious choice between - 4o (can search the web, use Canvas, evaluate Python server-side, generate images, but has no chain of thought) - o3-mini (web search, CoT, canvas, but no image generation) - o1 (CoT, maybe better than o3, but no canvas or web search and also no images) - Deep Research (very powerful, but I have…

For code it's actually quite good so far IME. Not quite as good as Gemini 2.5 Pro but much faster. I've integrated it into polychat.co if you want to try it out and compare with other models. I usually ask 2 to 5 models the same question there to reduce the model overload anxiety.

Re: GPT-4.1 in the API

#397

Post by cedws »

swyx wrote: post #203

don't miss that OAI also published a prompting guide WITH RECEIPTS for GPT 4.1 specifically for those building agents... with a new recommendation for: - telling the model to be persistent (+20%) - dont self-inject/parse toolcalls (+2%) - prompted planning (+4%) - JSON BAD - use XML or arxiv 2406.13121 (GDM format) - put instructions + user query at TOP -and- BOTTOM - bottom-only is VERY BAD - no evidence that ALL CA…

Why XML over JSON? Are they just saying that because XML is more tokens so they can make more money?

Re: GPT-4.1 in the API

#398

Post by jmtulloss »

modeless wrote: post #353
anotherpaulg wrote: post #284

Earlier quoted context omitted.

Aider author here. Based on some DMs with the Gemini team, they weren't aware that aider supports a "diff-fenced" edit format. And that it is specifically tuned to work well with Gemini models. So they didn't think to try it when they ran the aider benchmarks internally. Beyond that, I spend significant energy tuning aider to work well with top models. That is in fact the entire reason for aider's benchmark suite: to…

Thanks, that's interesting info. It seems to me that such tuning, while making Aider more useful, and making the benchmark useful in the specific context of deciding which model to use in Aider itself, reduces the value of the benchmark in evaluating overall model quality for use in other tools or contexts, as people use it for today. Models that get more tuning will outperform models that get less tuning, and existi…

I think you could argue the other side too... All of these models do better and worse with subtly different prompting that is non-obvious and unintuitive. Anybody using different models for "real work" are going to be tuning their prompts specifically to a model. Aider (without inside knowledge) can't possibly max out a given model's ability, but it can provide a reasonable approximation of what somebody can achieve with some effort.

Re: GPT-4.1 in the API

#399

Post by tdehnke »

I just wish they would start using human friendly names for them, and use a YY.rev version number so it's easier to know how new/old something is.

Broad Knowledge 25.1 Coder: Larger Problems 25.1 Coder: Line focused 25.1

Re: GPT-4.1 in the API

#400

Post by paradite »

999900000999 wrote: post #227

Have they implemented "I don't know" yet. I probably spend 100$ a month on AI coding, and it's great at small straightforward tasks. Drop it into a larger codebase and it'll get confused. Even if the same tool built it in the first place due to context limits. Then again, the way things are rapidly improving I suspect I can wait 6 months and they'll have a model that can do what I want.

Have you tried using a tool like 16x Prompt to send only relevant code to the model?

This helps the model to focus on a subset of codebase thst is relevant to the current task.

https://prompt.16x.engineer/

(I built it)

Post reply on HN