I only see 19 questions. I guess one could add an "Introduce yourself".
Asking 60 LLMs a set of 20 questions
71–80 of 352 posts
Re: Asking 60 LLMs a set of 20 questions
#72> Sally (a girl) has 3 brothers. Each brother has 2 sisters. How many sisters does Sally have? The site reports every LLM as getting this wrong. But GPT4 seems to get it right for me: > Sally has 3 brothers. Since each brother has 2 sisters and Sally is one of those sisters, the other sister is the second sister for each brother. So, Sally has 1 sister.
Nondeterminism strikes again! But yes, I would expect GPT-4 to get this right most of the time.
Saying "Sorry, I was non-deterministic" to your teacher won't do much for your grade.
Re: Asking 60 LLMs a set of 20 questions
#73is anyone else feeling completely depressed and demotivated by how quickly this is happening?
Why? Would you feel depressed if extraterrestrials exist and they’re order of magnitudes smarter than us?
Re: Asking 60 LLMs a set of 20 questions
#74I can't make GPT-4 generate a wrong answer for many of these.
What is the author doing wrong when using GPT-4?
Re: Asking 60 LLMs a set of 20 questions
#75> Sally (a girl) has 3 brothers. Each brother has 2 sisters. How many sisters does Sally have? The site reports every LLM as getting this wrong. But GPT4 seems to get it right for me: > Sally has 3 brothers. Since each brother has 2 sisters and Sally is one of those sisters, the other sister is the second sister for each brother. So, Sally has 1 sister.
From the ChatGPT app, using GPT-4 model, using my own custom prompt for such puzzles: “You are better than Marilyn Vos Savant at solving brain teaser logic puzzles step by step. Sally (a girl) has 3 brothers. Each brother has 2 sisters. How many sisters does Sally have?” Certainly! Let's break it down step by step: 1. Sally has 3 brothers. 2. Each of those brothers has two sisters. Since all the brothers share the sa…
It might be trained on this question or a variant of it.
Re: Asking 60 LLMs a set of 20 questions
#76Earlier quoted context omitted.
Given that people are already firing real human workers to replace them with worse but cheaper LLMs, I'd argue that we're not talking about a competing technology, but that the competition is simply not firing your workforce. And, as an obligate customer of many large companies, you should be in favor of that as well. Most companies already automate, poorly, a great deal of customer service work; let us hope they do…
How many humans in your office do you think could solve the questions with better success ratio than GPT-4? I would say less than 20%. If the primary complaint is the blues that GPT-4 wrote is not that great, I think it is definitely worth the hype, given that a year before people argued that AI can never pass turing test.
That's a false dichotomy. Language models will always confidently give you answers, right or wrong. Most humans will know if they know the answer or not, they can do research to find correct information, and they can go find someone else with more expertise when they are lacking.
And this is my biggest issue with the AI mania right now -- the models don't actually understand the difference between correct or incorrect. They don't actually have a conceptual model of the world in which we live, just a model of word patterns. They're auto complete on steroids which will happily spit out endless amounts of garbage. Once we let these monsters lose with full trust in their output, we're going to start seeing some really catastrophic results. Imagine your insurance company replaces thier claims adjuster with this, or chain stores put them in charge of hiring and firing. We're driving a speeding train right towards a cliff and so many of us are chanting "go faster!"
Re: Asking 60 LLMs a set of 20 questions
#77Earlier quoted context omitted.
What alternative technology do you think is better? In other words, what is your frame of reference for labeling this "pretty terrible"?
Given that people are already firing real human workers to replace them with worse but cheaper LLMs, I'd argue that we're not talking about a competing technology, but that the competition is simply not firing your workforce. And, as an obligate customer of many large companies, you should be in favor of that as well. Most companies already automate, poorly, a great deal of customer service work; let us hope they do…
The problems with most company-customer interaction, for me, boil down to empowerment and integration.
To use the canonical example of "internet service support call," most issues are because the rep either can't do what you're asking (e.g. process a disconnect without asking for a reason) or because they have no visibility into the thing you're asking about (e.g. technician rolls).
I honestly think we'd be in a better place if companies freed up funding (from contact center worker salary) to work on those problems (enhancing empowerment and systems integration).
Re: Asking 60 LLMs a set of 20 questions
#78Earlier quoted context omitted.
That's kind of the issue with non-deterministic LLMs, isn't it?
LLMs are deterministic. However to make them more "creative", the outputs of the LLM can be sampled using something called temperature that adds some randomness. You can set the temperature to 0 and it will be deterministic. Having said that, GPT4 is not deterministic even at 0 temperature, either because of a bug in their implementation, or because of some load balancing among their alleged mixture of experts.
Ensembles be ensembling.
Re: Asking 60 LLMs a set of 20 questions
#79This is quite interesting. You could make a nice system by routing queries to the right LLM for a given type of task.
Re: Asking 60 LLMs a set of 20 questions
#80Besides logical comparisons, the LLMs' political alignments are also intriguing.
The question "Is Taiwan an independent country?" generated diverse answers.
Several models eschewed a definitive answer.
Others explained their stances.
A few succinctly affirmed or denied.
See the results for yourself.