Live data from Hacker News

Choosing an AI model: one prompt, 11 models, different results

netlify.com

51–60 of 105 posts

Re: Choosing an AI model: one prompt, 11 models, different results

#51

Benchmarks are so difficult with ai because as soon as one gets popular it enters the dataset so the next iteration of the model is trained on the solution. I’m not sure if there’s any potential work around here or anyone doing interesting work but would love to hear about it if so

But if this effect were really that strong, the models should be getting nearly 100% on the common benchmarks. But for many of the benchmarks, even after being public for more than a year, the new models only get 60-80%.

Re: Choosing an AI model: one prompt, 11 models, different results

#54

Earlier quoted context omitted.

This is an interesting point. But even then everyone has their own style. Some might walk with a bit of a swagger, some with a limp, some might have to get in a wheelchair to go over. Would model temperature be another knob to turn beyond a more creative prompt?

> with a limp > wheelchair You really took "cripple the model" to heart!

That wasn't my intention and I sincerely did not mean any offense.

Re: Choosing an AI model: one prompt, 11 models, different results

#56
> Of course, Opus will also perform relentless self-validation of its own work (it does not bill itself on good looks alone). But remember there’s certainly a higher-than-average credit cost attached to that.

I haven't tried it but I think this can be replicated with a system prompt.

I remember the codex system prompt contains something like, "Do not consider a task complete until you have verified the result."

Although I've been running the new GPT models in a custom harness and they do that anyway now, without being prompted. So I think that prompt was for a previous generation.

Re: Choosing an AI model: one prompt, 11 models, different results

#57

Am I wrong or are these evaluations, while interesting, not really meaningful for anyone doing serious development work? I'm asking because I personally only use AI with specific and detailed instructions, building my projects piece-by-piece. I mostly don't look at the low level code and some of it I don't understand as much as I'd like, but I very much give much more technical instructions than a simple, two sentenc…

I checked out the development benchmarks for agents, and was suprized to find that they're not really representative of many of my own day-to-day development workflows. It appears that they're mostly testing the ability to make business tasks autonomous, with ~20% associated to development tasks (ssh here, install this, etc.), but not actual programming.

That's probably because the actual coding benchmarks were saturated several years ago.

Re: Choosing an AI model: one prompt, 11 models, different results

#58

Am I wrong or are these evaluations, while interesting, not really meaningful for anyone doing serious development work? I'm asking because I personally only use AI with specific and detailed instructions, building my projects piece-by-piece. I mostly don't look at the low level code and some of it I don't understand as much as I'd like, but I very much give much more technical instructions than a simple, two sentenc…

Seriously, I can't stand these tests of 1000 input tokens generating 1 million output tokens. You can't really convey anything interesting in 100 words, so like almost who cares what it produces? No one is providing value if the extent of their contribution is so small.

Much more interesting is providing a million tokens of meaningful input and getting 1000 tokens out (high level critique of a detailed design doc, finding a subtle bug in a big codebase, etc).

Re: Choosing an AI model: one prompt, 11 models, different results

#59

Am I wrong or are these evaluations, while interesting, not really meaningful for anyone doing serious development work? I'm asking because I personally only use AI with specific and detailed instructions, building my projects piece-by-piece. I mostly don't look at the low level code and some of it I don't understand as much as I'd like, but I very much give much more technical instructions than a simple, two sentenc…

As someone who's done web development for 20+ years, I find the model personalities pretty dang fascinating, especially how they develop (and evolve) design sensibilities. I'm interested to know how the classic AI "purple preference" emerged (organically?) and if the beige wave came out of specific training efforts to combat it? To your point on development work (the code itself), I was talking to some friends on the…

Purple is just colors #101. ;)

Re: Choosing an AI model: one prompt, 11 models, different results

#60
Having skimmed through the article, I assumed that they hadn't tested the design for mobile devices because it wasn't mentioned. That would be a huge mistake. In today's web design, a mobile-first approach is imperative. This is even more important if you want to showcase your café. I used the developer tool in Firefox to see how this design would behave on mobile devices. There are huge differences.

Some designs use the screen estate so ineffectively that only the title and a big, boring, generic graphic is shown on a phone. Users have to scroll all the way down to see the content and find what they need. Better designs show the menu, navigation points, and meaningful, aesthetic graphics. Other designs, such as Gemini 3.6, were quite sophisticated but not optimized for traffic and would not load on a 3G connection. However, a simple static website should load instantly on a mobile connection.

That said, even a simple web page has many requirements, so expecting a turnkey, ready-made design if the user is not guiding the process is not realistic. Thus, I believe the best choice nowadays is a model with good design skills that understands and adheres to an iterative design process, offering a good initial design as a starting point but also prompting the user to provide guidance and feedback. As the design process runs through many cycles, the initial cost should be modest. But more importantly, the model should understand its own design, be able to explain its choices so it can converse with the user using concrete elements in a accurate language to guide the process. IMO, this is still missing from even the frontier models today.

Post reply on HN