Asking 60 LLMs a set of 20 questions

Post by YetAnotherNick » Sun, Sep 10, 2023, 1:01 AM UTC

visarga wrote: post #161

YetAnotherNick wrote: post #57

Earlier quoted context omitted.

How many humans in your office do you think could solve the questions with better success ratio than GPT-4? I would say less than 20%. If the primary complaint is the blues that GPT-4 wrote is not that great, I think it is definitely worth the hype, given that a year before people argued that AI can never pass turing test.

LLMs might be better than random people at random tasks, and certainly memorised more trivia. But I never heard of a LLM surpassing subject experts in their field. On top, even the simplest task is unreliable - say, summarisation or translation. There is almost zero autonomy or ability to recover from unexpected situations.

Post by YetAnotherNick » Sun, Sep 10, 2023, 1:03 AM UTC

gtowey wrote: post #76

YetAnotherNick wrote: post #57

Earlier quoted context omitted.

How many humans in your office do you think could solve the questions with better success ratio than GPT-4? I would say less than 20%. If the primary complaint is the blues that GPT-4 wrote is not that great, I think it is definitely worth the hype, given that a year before people argued that AI can never pass turing test.

That's a false dichotomy. Language models will always confidently give you answers, right or wrong. Most humans will know if they know the answer or not, they can do research to find correct information, and they can go find someone else with more expertise when they are lacking. And this is my biggest issue with the AI mania right now -- the models don't actually understand the difference between correct or incorrec…

Post by YetAnotherNick » Sun, Sep 10, 2023, 1:48 AM UTC

detente18 wrote: post #68

YetAnotherNick wrote: post #62

Earlier quoted context omitted.

I think they messed up the templates.

Here's the template I'm using - https://github.com/BerriAI/litellm/blob/5ca8b23e22139a4f49bd... Anything I'm doing incorrectly?

Post by earthboundkid » Sun, Sep 10, 2023, 1:59 AM UTC

pixl97 wrote: post #258

earthboundkid wrote: post #239

Earlier quoted context omitted.

The thing with humans is they will say “I don’t remember how many syllables a haiku has” and “what the hell is kubernetes?” No LLM can reliably produce a haiku because their lexing process deprives them of reliable information about syllable counts. They should all say “I’m sorry, I can’t count syllables, but I’ll try my best anyway.” But the current models don’t do that because they were trained on texts by humans,…

Eh, it's not D&K gone berserk, it's what happens when you attempt to compress reality down to a single dimension (text). If you're doing a haiku, you will likely subvocalize it to ensure you're saying it correctly. It will be interesting when we get multimodal AI that can speak and listen to itself to detect things like this.

Post by awwaiid » Sun, Sep 10, 2023, 2:26 AM UTC

nojs wrote: post #12

> Sally (a girl) has 3 brothers. Each brother has 2 sisters. How many sisters does Sally have? The site reports every LLM as getting this wrong. But GPT4 seems to get it right for me: > Sally has 3 brothers. Since each brother has 2 sisters and Sally is one of those sisters, the other sister is the second sister for each brother. So, Sally has 1 sister.