Choosing an AI model: one prompt, 11 models, different results
1–10 of 105 posts
Re: Choosing an AI model: one prompt, 11 models, different results
#2> Build a one-page site for a neighbourhood coffee shop: opening hours, the address, a short menu and a photo. Nothing on it changes unless I edit it myself.
If that's the entire prompt, it's quite depressing how much alike these all look. I appreciate some of the details from the Opus 5 version, but I can't help but strongly feel the AI vibes emanating from that design.
Re: Choosing an AI model: one prompt, 11 models, different results
#3I always enjoy these comparisons between models, especially when they demonstrate the actual costs in addition to the outputs.
Re: Choosing an AI model: one prompt, 11 models, different results
#4Building ad-hoc evals is trivial these days. And you can place an LLM judge in front to disambiguate between output. e.g. I have Frigate gating a video feed so that an LLM can watch our home cameras and the labeling task takes a few minutes and then the evals run at fairly low cost https://wiki.roshangeorge.dev/w/images/4/4e/Screenshot_-_Eva...
This means that generic benchmarks and evals are sort of passé. There's no need to have them write "make a coffee shop page" or whatever. Instead, simply focus on the real problem you have and evaluate whether they solve that problem. You want to move along the price-performance frontier alone and the SOTA models are decidedly at the top of the performance curve but very expensive so they serve very well to produce the gold standard and to judge.
The change from prior to today is that benchmaxxing and per-token pricing allowing for evaluation point to the same direction: do not proxy results. Instead deploy the highest end model you have as a judge for others until you have statistical confidence in discrimination and move along the task-specific frontier.
Re: Choosing an AI model: one prompt, 11 models, different results
#5Interesting exploratory comparison, but I be cautious about treating it as a model benchmark
With only three runs per model, the results are highly sensitive to randomness
Re: Choosing an AI model: one prompt, 11 models, different results
#6I'd be curious to see Terra xhigh vs Sol low, only in that the visual languages are actually kinda different, that Terra work was a lot less "llm" feeling than a lot of the other designs, wonder if pumping up the effort would result in a more in-depth design but within that style.FWIW I enjoyed reading this way more than any usual benchmark posts we see.
Re: Choosing an AI model: one prompt, 11 models, different results
#7I've been doing a lot of benchmarking for a long time now with a number of local models for the purposes of spam filtering. The major observation is that there is a lot of variance in model performance. This should not be surprising, as these are probabilistic machines based on random numbers, so your performance will vary from run to run. But this also means that any sort of evaluation of benchmark with a sample size of 1 is essentially worthless for the purposes of model comparison.
In my benchmarks, I started insisting on having at least 5 runs.
This also makes me very suspicious of some of the posted benchmarks that compare models. If the results don't say how many benchmark runs were performed and what the variance was, there is really no basis for comparison. You are just guessing.
Re: Choosing an AI model: one prompt, 11 models, different results
#8Am I wrong or are these evaluations, while interesting, not really meaningful for anyone doing serious development work?
I'm asking because I personally only use AI with specific and detailed instructions, building my projects piece-by-piece.
I mostly don't look at the low level code and some of it I don't understand as much as I'd like, but I very much give much more technical instructions than a simple, two sentence prompt.
So, it seems to me this "oneshot from a simple prompt" eval is fairly meaningless when it comes to model evaluation itself, as it is in no way representative of real world application.
This would be more something for "vibe coders", people with little to no programming background wanting a website?
Re: Choosing an AI model: one prompt, 11 models, different results
#9> Build a one-page site for a neighbourhood coffee shop: opening hours, the address, a short menu and a photo. Nothing on it changes unless I edit it myself. If that's the entire prompt, it's quite depressing how much alike these all look. I appreciate some of the details from the Opus 5 version, but I can't help but strongly feel the AI vibes emanating from that design.
Without more knowledge about the technical aspect, it could be a good thing that they're all so similar. If you tell humans to go from their kitchen to their bedroom, they all stand up and walk the same way. Nobody decides to crab walk. Maybe you'd have to cripple the model in some way to do that.
On the other hand, if you want something different with LLMs, all it takes is a few more words of creative flair in the prompt.
Re: Choosing an AI model: one prompt, 11 models, different results
#10> Build a one-page site for a neighbourhood coffee shop: opening hours, the address, a short menu and a photo. Nothing on it changes unless I edit it myself. If that's the entire prompt, it's quite depressing how much alike these all look. I appreciate some of the details from the Opus 5 version, but I can't help but strongly feel the AI vibes emanating from that design.
It’s always been interesting to see how similar output is across ostensibly very different models. I remember testing short story writing in the early days and having the models all choose the same niche topic across e.g GPT, Llama, Phi, Claude, etc