Benchmarks are so difficult with ai because as soon as one gets popular it enters the dataset so the next iteration of the model is trained on the solution. I’m not sure if there’s any potential work around here or anyone doing interesting work but would love to hear about it if so
Choosing an AI model: one prompt, 11 models, different results
41–50 of 105 posts
Re: Choosing an AI model: one prompt, 11 models, different results
#42This seems to be the easiest way to get on HN front page:
Come up with an arbitrary test, let a bunch of LLMs work on it. Make some very subjective judgement about the result...
Re: Choosing an AI model: one prompt, 11 models, different results
#43Earlier quoted context omitted.
To see the hours, get a sense of the menu. There are many matcha shops by me for instance and my wife likes to check the specials before choosing which one to go to.
Doesn't most of this show up on google anyway?
Where do you think Google and co. source that data from? Having an actual, searchable, up to date menu is, just for a11y and accuracy far more valuable than someone’s poorly lit, low res upload of an photo taken from the menu circa 2019, just to name one advantage. Have found data from restaurants without their own webpages on Maps utterly inaccurate, even suggesting some that have been shut down for months to where there is clear value. Then there is restaurants closing for holidays, illness, etc. and the ability to make reservations without calling.
Whether these need their own over an Astro template is another topic, but having a webpage at all is incredibly valuable.
Re: Choosing an AI model: one prompt, 11 models, different results
#44Earlier quoted context omitted.
Without more knowledge about the technical aspect, it could be a good thing that they're all so similar. If you tell humans to go from their kitchen to their bedroom, they all stand up and walk the same way. Nobody decides to crab walk. Maybe you'd have to cripple the model in some way to do that. On the other hand, if you want something different with LLMs, all it takes is a few more words of creative flair in the p…
This is an interesting point. But even then everyone has their own style. Some might walk with a bit of a swagger, some with a limp, some might have to get in a wheelchair to go over. Would model temperature be another knob to turn beyond a more creative prompt?
> with a limp
> wheelchair
You really took "cripple the model" to heart!
Re: Choosing an AI model: one prompt, 11 models, different results
#45Earlier quoted context omitted.
> Nobody is saying, "man I would love to go to this coffee shop but I can't find their website". Me, I'm saying that, and I've skipped going to coffee shops and restaurants because they don't have a website, just a fucking Facebook page . I don't use Meta products and can't see their page if I'm not logged into an account I don't have, so I do what the business owner intended: I go fuck myself and get coffee somewher…
But... why do you want to look up a coffee shop online before going there? Honestly have never heard anyone say this in my life before
Why does anyone look up anything before going there? Maybe no one should have a website, everyone should go check everything out themselves in person like the telegraph hasn't been invented.
Re: Choosing an AI model: one prompt, 11 models, different results
#46[deleted]
Re: Choosing an AI model: one prompt, 11 models, different results
#47Finally some really useful apples-to-apples comparisons rather than endless benchmarkmaxxing and anecdata. I want to see this test run again for the open-weights models that fit into 128GB combined RAM. Like how does Muse-glimmer compare with qwen3.6-35B?
Re: Choosing an AI model: one prompt, 11 models, different results
#48I like the output of the cheaper/older models better, surprisingly, say Gemini 3.1 or DeepSeek V4. They're mostly no frills and just text, and ironically look less AI-generated to me because of the lack of hip slogans and graphics.
Definitely closer to what I'd want for my own site, but I don't claim to know what people want from a website for a coffeeshop.
Re: Choosing an AI model: one prompt, 11 models, different results
#49> Build a one-page site for a neighbourhood coffee shop: opening hours, the address, a short menu and a photo. Nothing on it changes unless I edit it myself. If that's the entire prompt, it's quite depressing how much alike these all look. I appreciate some of the details from the Opus 5 version, but I can't help but strongly feel the AI vibes emanating from that design.
A user, even without much design background, could ask the AI in a follow up prompt to change a few key parameters like overall color and differentiate it more. Even taking a sample image from a design magazine or blog and saying "incorporate some of the design elements here" would probably work great to diverge it.
Re: Choosing an AI model: one prompt, 11 models, different results
#50this is fun!