Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Choosing an AI model: one prompt, 11 models, different results

netlify.com

Post reply on HN
Print view View original

31–40 of 105 posts

Re: Choosing an AI model: one prompt, 11 models, different results

#31

Post by seamlessdev »

runtime_terror wrote: post #21
toddmorey wrote: post #17

Earlier quoted context omitted.

As someone who's done web development for 20+ years, I find the model personalities pretty dang fascinating, especially how they develop (and evolve) design sensibilities. I'm interested to know how the classic AI "purple preference" emerged (organically?) and if the beige wave came out of specific training efforts to combat it? To your point on development work (the code itself), I was talking to some friends on the…

I'd assume it comes from Tailwind boilerplate/template sites as it seemed almost all of them were purple at the time

Definitely. It was already a trend before LLMs exploded. And the whole "all websites look the same" has been a thing since at least Bootstrap times.

Re: Choosing an AI model: one prompt, 11 models, different results

#33

Post by s4i »

In my opinion, this kind of a benchmark doesn't tell much about the models' capabilities on normal software development tasks. It's fun to look at the differences in the output of course, but how often would anyone prompt with very brief instructions, without even hinting the model about caring about any of the details in the outcome nor the implementation?

When there are implicit boundaries, negotiable tradeoffs, taste, whatever, in the mix, then the differences in the model capabilities become way more interesting.

Re: Choosing an AI model: one prompt, 11 models, different results

#34

Post by mym1990 »

tantalor wrote: post #12
isqueiros wrote: post #2

> Build a one-page site for a neighbourhood coffee shop: opening hours, the address, a short menu and a photo. Nothing on it changes unless I edit it myself. If that's the entire prompt, it's quite depressing how much alike these all look. I appreciate some of the details from the Opus 5 version, but I can't help but strongly feel the AI vibes emanating from that design.

It's also kind of pointless. Why does a coffee shop need a website anyway? Nobody is saying, "man I would love to go to this coffee shop but I can't find their website". If all you want is "opening hours, the address, a short menu and a photo" there are easier and cheaper ways to do that.

Why does your brain instantly go to "whats the cheapest and easiest way to do things", instead of "whats creative, fun, interactive, usable, accessible, etc...".

This is the reason the world is slowly becoming a boring ass place and the internet basically consists of 3 websites that are all trying to suck your soul dry.

If I ran a business, a website that stands out from the crowd and provides a good experience would be in my top 3 things to do.

Re: Choosing an AI model: one prompt, 11 models, different results

#36

Post by mym1990 »

altmanaltman wrote: post #29
skinfaxi wrote: post #26

Earlier quoted context omitted.

To see the hours, get a sense of the menu. There are many matcha shops by me for instance and my wife likes to check the specials before choosing which one to go to.

Doesn't most of this show up on google anyway?

It sounds like you know everything about the internet and people's preferences, why ask?

Re: Choosing an AI model: one prompt, 11 models, different results

#38

Post by mym1990 »

altmanaltman wrote: post #23
miyoji wrote: post #16

Earlier quoted context omitted.

> Nobody is saying, "man I would love to go to this coffee shop but I can't find their website". Me, I'm saying that, and I've skipped going to coffee shops and restaurants because they don't have a website, just a fucking Facebook page . I don't use Meta products and can't see their page if I'm not logged into an account I don't have, so I do what the business owner intended: I go fuck myself and get coffee somewher…

But... why do you want to look up a coffee shop online before going there? Honestly have never heard anyone say this in my life before

It amazes me how close minded people have gotten. Do you really have that hard of a time believing there is a use case for a human looking up information on a coffee shop, on the internet? Oh, and god forbid the shop owner actually wants to provide it!

Re: Choosing an AI model: one prompt, 11 models, different results

#39

Post by jamiedumont »

isqueiros wrote: post #2

> Build a one-page site for a neighbourhood coffee shop: opening hours, the address, a short menu and a photo. Nothing on it changes unless I edit it myself. If that's the entire prompt, it's quite depressing how much alike these all look. I appreciate some of the details from the Opus 5 version, but I can't help but strongly feel the AI vibes emanating from that design.

I trialed Claude for a sorely needed redesign on my website. I was quite impressed with the results, but decided to hang fire on deploying it. Cue my surprise when I just found the same design applied to a coffee shop!

The similarities are beyond coincidence, to the point I'll be scrapping Claude's version of the redesign.

Re: Choosing an AI model: one prompt, 11 models, different results

#40

Post by michaelbuckbee »

Systemerror7A69 wrote: post #8

Am I wrong or are these evaluations, while interesting, not really meaningful for anyone doing serious development work? I'm asking because I personally only use AI with specific and detailed instructions, building my projects piece-by-piece. I mostly don't look at the low level code and some of it I don't understand as much as I'd like, but I very much give much more technical instructions than a simple, two sentenc…

The eval world is split into:

1. Long form task based examinations like this that test the ability of the model+harness to remain on task, tool calling, overall effectiveness and taste.

2. More direct 1:1 and qualitative comparisons that you might get with a tool like https://evvl.ai/ - which also uses OpenRouter and does similar one off model comparisons (or lets you use it as a MCP from your dev env to be like: "take the prompt from this loop and try it against these other models")

Post reply on HN