Live data from Hacker News

Choosing an AI model: one prompt, 11 models, different results

netlify.com

81–90 of 105 posts

Re: Choosing an AI model: one prompt, 11 models, different results

#81

Earlier quoted context omitted.

> with a limp > wheelchair You really took "cripple the model" to heart!

That wasn't my intention and I sincerely did not mean any offense.

I don't think they took offense; more likely what they meant was "limping" and "wheelchair" aren't considered normal ways to get around, because there's an abnormal condition that led to said ways of movement. Similarly LLMs with similar capabilities operating in a normal way should produce relatively similar output.

Re: Choosing an AI model: one prompt, 11 models, different results

#82
post #43

Earlier quoted context omitted.

Doesn't most of this show up on google anyway?

Where do you think Google and co. source that data from? Having an actual, searchable, up to date menu is, just for a11y and accuracy far more valuable than someone’s poorly lit, low res upload of an photo taken from the menu circa 2019, just to name one advantage. Have found data from restaurants without their own webpages on Maps utterly inaccurate, even suggesting some that have been shut down for months to where…

If you are a business, you can set up google's business feature and provide the data and high-quality images. That is basic digital presence hygeine. If a shop is refusing to do that, they will definitely not care about a website

Re: Choosing an AI model: one prompt, 11 models, different results

#83
post #36

Earlier quoted context omitted.

Doesn't most of this show up on google anyway?

It sounds like you know everything about the internet and people's preferences, why ask?

You seem to ask rhetorical questions that are not related to the topic and are just personal attacks, why?

Re: Choosing an AI model: one prompt, 11 models, different results

#84

Earlier quoted context omitted.

But... why do you want to look up a coffee shop online before going there? Honestly have never heard anyone say this in my life before

When I travel I have limited time, and if e.g. the coffee shop doesn't have the hallmarks of good coffee, I will save my time and find one that does

So you compare by visiting every website of each coffee shop? And that saves you time?

Re: Choosing an AI model: one prompt, 11 models, different results

#85
post #45

Earlier quoted context omitted.

But... why do you want to look up a coffee shop online before going there? Honestly have never heard anyone say this in my life before

Why does anyone look up anything before going there? Maybe no one should have a website, everyone should go check everything out themselves in person like the telegraph hasn't been invented.

No, we should look up everything before we go anywhere. Everyone should have their own website and constantly update and maintain it everyday even though most of your customers never visit and just use google which can be set up without having to spend and maintain on a website which many chose to do.

And why is it so wrong to go to a place to drink coffee without researching it in the first place?

Re: Choosing an AI model: one prompt, 11 models, different results

#87

Am I wrong or are these evaluations, while interesting, not really meaningful for anyone doing serious development work? I'm asking because I personally only use AI with specific and detailed instructions, building my projects piece-by-piece. I mostly don't look at the low level code and some of it I don't understand as much as I'd like, but I very much give much more technical instructions than a simple, two sentenc…

Seriously, I can't stand these tests of 1000 input tokens generating 1 million output tokens. You can't really convey anything interesting in 100 words, so like almost who cares what it produces? No one is providing value if the extent of their contribution is so small. Much more interesting is providing a million tokens of meaningful input and getting 1000 tokens out (high level critique of a detailed design doc, fi…

I think both can be interesting, but surely the 100 words will lead to a lot of variability in the exact shape of the result because so much of the spec will be left up to the LLM. However, it is still interesting to judge the quality of the result and how self-consistent it is. Not by looking at the actual webpage, but by looking at the code and whether it is nicely organized, makes proper use of abstractions, and in general doesn't feel like spaghetti.

Re: Choosing an AI model: one prompt, 11 models, different results

#88

Am I wrong or are these evaluations, while interesting, not really meaningful for anyone doing serious development work? I'm asking because I personally only use AI with specific and detailed instructions, building my projects piece-by-piece. I mostly don't look at the low level code and some of it I don't understand as much as I'd like, but I very much give much more technical instructions than a simple, two sentenc…

Seriously, I can't stand these tests of 1000 input tokens generating 1 million output tokens. You can't really convey anything interesting in 100 words, so like almost who cares what it produces? No one is providing value if the extent of their contribution is so small. Much more interesting is providing a million tokens of meaningful input and getting 1000 tokens out (high level critique of a detailed design doc, fi…

Great - can you show us some examples of these? Every time this topic comes up here it’s full of people saying “you’re using it wrong” and apparently it should just be intuitive. Yet all the material is like this.

Re: Choosing an AI model: one prompt, 11 models, different results

#89
post #15

Am I wrong or are these evaluations, while interesting, not really meaningful for anyone doing serious development work? I'm asking because I personally only use AI with specific and detailed instructions, building my projects piece-by-piece. I mostly don't look at the low level code and some of it I don't understand as much as I'd like, but I very much give much more technical instructions than a simple, two sentenc…

I'm repeatedly noticing that people working at big ai and tech companies are surprisingly not that... good... at using ai? It's like theyre doing a plausible thing to get something done and calling it a day

Can you share some posts of good examples of prompts and comparisons?

Re: Choosing an AI model: one prompt, 11 models, different results

#90
post #7

I've been doing a lot of benchmarking for a long time now with a number of local models for the purposes of spam filtering. The major observation is that there is a lot of variance in model performance. This should not be surprising, as these are probabilistic machines based on random numbers, so your performance will vary from run to run. But this also means that any sort of evaluation of benchmark with a sample siz…

Have you published any of your results? I am quite curious.
Post reply on HN