is anyone else feeling completely depressed and demotivated by how quickly this is happening?
Asking 60 LLMs a set of 20 questions
41–50 of 352 posts
Re: Asking 60 LLMs a set of 20 questions
#42> Sally (a girl) has 3 brothers. Each brother has 2 sisters. How many sisters does Sally have? The site reports every LLM as getting this wrong. But GPT4 seems to get it right for me: > Sally has 3 brothers. Since each brother has 2 sisters and Sally is one of those sisters, the other sister is the second sister for each brother. So, Sally has 1 sister.
That's kind of the issue with non-deterministic LLMs, isn't it?
Re: Asking 60 LLMs a set of 20 questions
#43Great work. This really gives an insight on how much things change when you go up in parameter count - not always, but you can see results change.
How did you run the queries against these engines? Did you host the inference engines yourself or did you have to sign up for services. If there was a way to supplement each LLM with additional data I can see this being a useful service for companies who are investigating ML in various facets of their business.
Re: Asking 60 LLMs a set of 20 questions
#44Despite the hype about LLMs, many of the answers are pretty terrible. The 12-bar blues progressions seem mostly clueless. The question is will any of these ever get significantly better with time, or are they mostly going to stagnate?
Re: Asking 60 LLMs a set of 20 questions
#45Despite the hype about LLMs, many of the answers are pretty terrible. The 12-bar blues progressions seem mostly clueless. The question is will any of these ever get significantly better with time, or are they mostly going to stagnate?
What alternative technology do you think is better?
In other words, what is your frame of reference for labeling this "pretty terrible"?
Re: Asking 60 LLMs a set of 20 questions
#46One of the jokes: "I'm going on vacation, but I'm not sure if I should take my laptop. It's such a pain in the neck to lug it around with me."
User
I feel like this bot mocking us
Re: Asking 60 LLMs a set of 20 questions
#47Are these LLMs deterministic or is this comparison rather useless?
I'm voting pretty useless -- I've dorked around with Llama and CodeLlama a bit, and wow, the differences between runs are pretty substantial.
Re: Asking 60 LLMs a set of 20 questions
#48> Sally (a girl) has 3 brothers. Each brother has 2 sisters. How many sisters does Sally have? The site reports every LLM as getting this wrong. But GPT4 seems to get it right for me: > Sally has 3 brothers. Since each brother has 2 sisters and Sally is one of those sisters, the other sister is the second sister for each brother. So, Sally has 1 sister.
GPT 4 and another LLM have given the right answer only after adding "Let's think step by step." to the original prompt.
With the simpler prompt, all the answers were wrong, most of them ridiculously wrong.
Re: Asking 60 LLMs a set of 20 questions
#49Despite the hype about LLMs, many of the answers are pretty terrible. The 12-bar blues progressions seem mostly clueless. The question is will any of these ever get significantly better with time, or are they mostly going to stagnate?
What alternative technology do you think is better? In other words, what is your frame of reference for labeling this "pretty terrible"?
Humans. After all, LLMs are designed to reason equal to or better than humans.
Re: Asking 60 LLMs a set of 20 questions
#50Has anyone looked through all the responses and chosen any winners?
I took at the look at the SVG smilies, only GPT gets it right (https://benchmarks.llmonitor.com/svg). You can preview the SVGs drawn by the LLMs by pasting this into the browser console
document.querySelectorAll("td pre").forEach((node) => { let code = node.textContent; node.insertAdjacentHTML('afterend', code) })
Or take a look at my screenshot: https://i.ibb.co/Kw0kp58/Screenshot-2023-09-09-at-17-15-20-h...
