Asking 60 LLMs a set of 20 questions • Hacker News Board

Re: Asking 60 LLMs a set of 20 questions

#81

Post by facu17y » Sat, Sep 09, 2023, 4:00 PM UTC

peyton wrote: post #59

0xDEF wrote: post #56

Earlier quoted context omitted.

I asked GPT-4 through both ChatGPT and the API and it responded correctly without any fiddling. I keep seeing comments and posts on HN that significantly downplay GPT-4's capabilities. Are people actually using GPT-4 or are they using a 3rd party service that claims to be GPT-4? I got: >Sally has 3 brothers, and each of those brothers has 2 sisters. One of those sisters is Sally herself, and the other one is Sally's…

Here’s what ChatGPT GPT-4 gives me: > Sally has 2 sisters. Each of her 3 brothers has 2 sisters, and those sisters would be Sally and her 2 sisters.

Every now and then GPT4 outputs a wrong answer. It's impossible to build a reliable product on top of GPT4 that is not a simple chat bot.

Re: Asking 60 LLMs a set of 20 questions

#82

Post by famouswaffles » Sat, Sep 09, 2023, 4:00 PM UTC

gtowey wrote: post #76

YetAnotherNick wrote: post #57

Earlier quoted context omitted.

How many humans in your office do you think could solve the questions with better success ratio than GPT-4? I would say less than 20%. If the primary complaint is the blues that GPT-4 wrote is not that great, I think it is definitely worth the hype, given that a year before people argued that AI can never pass turing test.

That's a false dichotomy. Language models will always confidently give you answers, right or wrong. Most humans will know if they know the answer or not, they can do research to find correct information, and they can go find someone else with more expertise when they are lacking. And this is my biggest issue with the AI mania right now -- the models don't actually understand the difference between correct or incorrec…

>Most humans will know if they know the answer or not,

No they won't.

>they can go find someone else with more expertise when they are lacking.

They can but they often don't.

>the models don't actually understand the difference between correct or incorrect.

They certainly do

https://imgur.com/a/3gYel9r

Re: Asking 60 LLMs a set of 20 questions

#83

Post by kordlessagain » Sat, Sep 09, 2023, 4:01 PM UTC

dinkleberg wrote: post #79

This is quite interesting. You could make a nice system by routing queries to the right LLM for a given type of task.

Right. By logging all of this, then vectorizing, then evaluating using some type of transfer knowledge (classify this response and it's accuracy), one could build up a routing table of likely candidates for building an ensemble to evaluate the correct answer.

Re: Asking 60 LLMs a set of 20 questions

#84

Post by falcor84 » Sat, Sep 09, 2023, 4:02 PM UTC

I love how Alpaca (7B) added a sarcasm tag to its (incorrect) response to the leap year question.

    def print_leap_years():
        for i in range(20):
            if i % 400 == 0:
                print(i)
        print("End")

    print_leap_years()

Re: Asking 60 LLMs a set of 20 questions

#85

Post by PUSH_AX » Sat, Sep 09, 2023, 4:02 PM UTC

dinkleberg wrote: post #79

This is quite interesting. You could make a nice system by routing queries to the right LLM for a given type of task.

Like having a mixture of experts!

Re: Asking 60 LLMs a set of 20 questions

#86

Post by civilitty » Sat, Sep 09, 2023, 4:03 PM UTC

facu17y wrote: post #75

Terretta wrote: post #40

Earlier quoted context omitted.

From the ChatGPT app, using GPT-4 model, using my own custom prompt for such puzzles: “You are better than Marilyn Vos Savant at solving brain teaser logic puzzles step by step. Sally (a girl) has 3 brothers. Each brother has 2 sisters. How many sisters does Sally have?” Certainly! Let's break it down step by step: 1. Sally has 3 brothers. 2. Each of those brothers has two sisters. Since all the brothers share the sa…

It might be trained on this question or a variant of it.

It's certainly RLHFed. All of the logic puzzles I use for evaluation that used to fail months ago now pass no problem and I've even had a hard time modifying them to fail.

Re: Asking 60 LLMs a set of 20 questions

#87

Post by 101011 » Sat, Sep 09, 2023, 4:04 PM UTC

redox99 wrote: post #69

mmcwilliams wrote: post #42

Earlier quoted context omitted.

That's kind of the issue with non-deterministic LLMs, isn't it?

LLMs are deterministic. However to make them more "creative", the outputs of the LLM can be sampled using something called temperature that adds some randomness. You can set the temperature to 0 and it will be deterministic. Having said that, GPT4 is not deterministic even at 0 temperature, either because of a bug in their implementation, or because of some load balancing among their alleged mixture of experts.

This was interesting to me, so I dug a bit further. This gives a bit more context behind why: https://community.openai.com/t/observing-discrepancy-in-comp...

Quote below:

Even with a greedy decoding strategy, small discrepancies regarding floating point operations lead to divergent generations. In simpler terms: when the top-two tokens have very similar log-probs, there’s a non-zero probability of choosing the least probable one due to the finite number of digits that you’re using for multiplying probs and storing them.

It should also be noted that, as the decoding occurs in an autoregressive way, once you have picked a different token the whole generated sequence will diverge, as this choice affects to the probability of generating every subsequent token.

Re: Asking 60 LLMs a set of 20 questions

#88

Post by ComputerGuru » Sat, Sep 09, 2023, 4:05 PM UTC

smilingemoji wrote: post #50

sixQuarks wrote: post #9

Has anyone looked through all the responses and chosen any winners?

I took at the look at the SVG smilies, only GPT gets it right ( https://benchmarks.llmonitor.com/svg ). You can preview the SVGs drawn by the LLMs by pasting this into the browser console document.querySelectorAll("td pre").forEach((node) => { let code = node.textContent; node.insertAdjacentHTML('afterend', code) }) Or take a look at my screenshot: https://i.ibb.co/Kw0kp58/Screenshot-2023-09-09-at-17-15-20-h...

Clever use of the dev tools console. I think you’re being a little uncharitable, the prompt said “smiley” and not “smiling emoji” and “smiley” was once upon a time used for the whole slew of emoji. With that in mind, several others also pass.

Re: Asking 60 LLMs a set of 20 questions

#89

Post by slama » Sat, Sep 09, 2023, 4:06 PM UTC

falcor84 wrote: post #84

I love how Alpaca (7B) added a sarcasm tag to its (incorrect) response to the leap year question. def print_leap_years(): for i in range(20): if i % 400 == 0: print(i) print("End") print_leap_years()

That's a stop token, not a sarcasm tag

Re: Asking 60 LLMs a set of 20 questions

#90

Post by sirk390 » Sat, Sep 09, 2023, 4:10 PM UTC

salil999 wrote: post #49

smokel wrote: post #45

Earlier quoted context omitted.

What alternative technology do you think is better? In other words, what is your frame of reference for labeling this "pretty terrible"?

Humans. After all, LLMs are designed to reason equal to or better than humans.

Humans are pretty bad at these questions. Even with the simplest questions like "Sally (a girl) has 3 brothers. Each brother has 2 sisters. How many sisters does Sally have?" I think that a lot of people will give an incorrect answer. And for questions like "Argue for and against the use of kubernetes in the style of a haiku", 99.99% will not be able to do it.