Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Choosing an AI model: one prompt, 11 models, different results

netlify.com

Post reply on HN
Print view View original

11–20 of 105 posts

Re: Choosing an AI model: one prompt, 11 models, different results

#11

Post by horsawlarway »

Approaching this from the perspective of a potential customer and not a designer - I find that I actually like the smaller/open model output quite a bit more.

Kimi, GLM, and Deepseek all absolutely run away with the "Can I quickly read the menu and find the address" challenge.

Most of the rest of the pages are stylistic, but hard to parse.

If I were in a car on a mobile phone trying to find the address of the place to meet a friend for coffee... I don't want a bunch of fluff and stylistic design that makes it hard to parse the information on the site.

Re: Choosing an AI model: one prompt, 11 models, different results

#12

Post by tantalor »

isqueiros wrote: post #2

> Build a one-page site for a neighbourhood coffee shop: opening hours, the address, a short menu and a photo. Nothing on it changes unless I edit it myself. If that's the entire prompt, it's quite depressing how much alike these all look. I appreciate some of the details from the Opus 5 version, but I can't help but strongly feel the AI vibes emanating from that design.

It's also kind of pointless. Why does a coffee shop need a website anyway? Nobody is saying, "man I would love to go to this coffee shop but I can't find their website".

If all you want is "opening hours, the address, a short menu and a photo" there are easier and cheaper ways to do that.

Re: Choosing an AI model: one prompt, 11 models, different results

#15

Post by 217 »

Systemerror7A69 wrote: post #8

Am I wrong or are these evaluations, while interesting, not really meaningful for anyone doing serious development work? I'm asking because I personally only use AI with specific and detailed instructions, building my projects piece-by-piece. I mostly don't look at the low level code and some of it I don't understand as much as I'd like, but I very much give much more technical instructions than a simple, two sentenc…

I'm repeatedly noticing that people working at big ai and tech companies are surprisingly not that... good... at using ai? It's like theyre doing a plausible thing to get something done and calling it a day

Re: Choosing an AI model: one prompt, 11 models, different results

#16

Post by miyoji »

tantalor wrote: post #12
isqueiros wrote: post #2

> Build a one-page site for a neighbourhood coffee shop: opening hours, the address, a short menu and a photo. Nothing on it changes unless I edit it myself. If that's the entire prompt, it's quite depressing how much alike these all look. I appreciate some of the details from the Opus 5 version, but I can't help but strongly feel the AI vibes emanating from that design.

It's also kind of pointless. Why does a coffee shop need a website anyway? Nobody is saying, "man I would love to go to this coffee shop but I can't find their website". If all you want is "opening hours, the address, a short menu and a photo" there are easier and cheaper ways to do that.

> Nobody is saying, "man I would love to go to this coffee shop but I can't find their website".

Me, I'm saying that, and I've skipped going to coffee shops and restaurants because they don't have a website, just a fucking Facebook page. I don't use Meta products and can't see their page if I'm not logged into an account I don't have, so I do what the business owner intended: I go fuck myself and get coffee somewhere else.

Re: Choosing an AI model: one prompt, 11 models, different results

#17

Post by toddmorey »

Systemerror7A69 wrote: post #8

Am I wrong or are these evaluations, while interesting, not really meaningful for anyone doing serious development work? I'm asking because I personally only use AI with specific and detailed instructions, building my projects piece-by-piece. I mostly don't look at the low level code and some of it I don't understand as much as I'd like, but I very much give much more technical instructions than a simple, two sentenc…

As someone who's done web development for 20+ years, I find the model personalities pretty dang fascinating, especially how they develop (and evolve) design sensibilities.

I'm interested to know how the classic AI "purple preference" emerged (organically?) and if the beige wave came out of specific training efforts to combat it?

To your point on development work (the code itself), I was talking to some friends on the Google Chrome team about any research understanding the model's preferences around framework ergonomics and abilities to properly implement core web standards for given tasks. I think that would be super fascinating.

Re: Choosing an AI model: one prompt, 11 models, different results

#18

Post by pistoriusp »

Systemerror7A69 wrote: post #8

Am I wrong or are these evaluations, while interesting, not really meaningful for anyone doing serious development work? I'm asking because I personally only use AI with specific and detailed instructions, building my projects piece-by-piece. I mostly don't look at the low level code and some of it I don't understand as much as I'd like, but I very much give much more technical instructions than a simple, two sentenc…

I checked out the development benchmarks for agents, and was suprized to find that they're not really representative of many of my own day-to-day development workflows.

It appears that they're mostly testing the ability to make business tasks autonomous, with ~20% associated to development tasks (ssh here, install this, etc.), but not actual programming.

Re: Choosing an AI model: one prompt, 11 models, different results

#20

Post by toddmorey »

jwr wrote: post #7

I've been doing a lot of benchmarking for a long time now with a number of local models for the purposes of spam filtering. The major observation is that there is a lot of variance in model performance. This should not be surprising, as these are probabilistic machines based on random numbers, so your performance will vary from run to run. But this also means that any sort of evaluation of benchmark with a sample siz…

Yeah the difference in token usage across difference models they found in the article was broader than I expected, but I was sort of floored by the variance in token usage for the same prompt (run multiple times) with the same model.
Post reply on HN