51–60 of 352 posts
Despite the hype about LLMs, many of the answers are pretty terrible. The 12-bar blues progressions seem mostly clueless. The question is will any of these ever get significantly better with time, or are they mostly going to stagnate?
I expect additional advances at some point in the future.
Despite the hype about LLMs, many of the answers are pretty terrible. The 12-bar blues progressions seem mostly clueless. The question is will any of these ever get significantly better with time, or are they mostly going to stagnate?
What alternative technology do you think is better? In other words, what is your frame of reference for labeling this "pretty terrible"?
And, as an obligate customer of many large companies, you should be in favor of that as well. Most companies already automate, poorly, a great deal of customer service work; let us hope they do not force us to interact with these deeply useless things as well.
> Sally has 0 sisters. The question provides no information about Sally having any sisters herself.
It isn't entirely wrong, is it?
> Sally (a girl) has 3 brothers. Each brother has 2 sisters. How many sisters does Sally have? The site reports every LLM as getting this wrong. But GPT4 seems to get it right for me: > Sally has 3 brothers. Since each brother has 2 sisters and Sally is one of those sisters, the other sister is the second sister for each brother. So, Sally has 1 sister.
GPT 4 and another LLM have given the right answer only after adding "Let's think step by step." to the original prompt. With the simpler prompt, all the answers were wrong, most of them ridiculously wrong.
Ultimately I feel it is fairer to benchmark llm’s by what they can be prompted into. After all, we let people carefully work through a problem during exams so it seems fair to hold llm’s to the same standard.
> Sally (a girl) has 3 brothers. Each brother has 2 sisters. How many sisters does Sally have? The site reports every LLM as getting this wrong. But GPT4 seems to get it right for me: > Sally has 3 brothers. Since each brother has 2 sisters and Sally is one of those sisters, the other sister is the second sister for each brother. So, Sally has 1 sister.
Also, MPT 7B gets it right over half the time. I've been testing every new LLM with that question. Also, I tend to include mention in the question that all siblings are from the same two parents to preclude half-siblings because half my friends have half-siblings from both sides scattered across the country; so the wrong answers actually do tend to apply to them sometimes.
We should pay more attention to data contamination when using popular prompts for testing.
> Sally (a girl) has 3 brothers. Each brother has 2 sisters. How many sisters does Sally have? The site reports every LLM as getting this wrong. But GPT4 seems to get it right for me: > Sally has 3 brothers. Since each brother has 2 sisters and Sally is one of those sisters, the other sister is the second sister for each brother. So, Sally has 1 sister.
GPT 4 and another LLM have given the right answer only after adding "Let's think step by step." to the original prompt. With the simpler prompt, all the answers were wrong, most of them ridiculously wrong.
I keep seeing comments and posts on HN that significantly downplay GPT-4's capabilities. Are people actually using GPT-4 or are they using a 3rd party service that claims to be GPT-4?
I got:
>Sally has 3 brothers, and each of those brothers has 2 sisters. One of those sisters is Sally herself, and the other one is Sally's sister. So, Sally has 1 sister.
Earlier quoted context omitted.
What alternative technology do you think is better? In other words, what is your frame of reference for labeling this "pretty terrible"?
Given that people are already firing real human workers to replace them with worse but cheaper LLMs, I'd argue that we're not talking about a competing technology, but that the competition is simply not firing your workforce. And, as an obligate customer of many large companies, you should be in favor of that as well. Most companies already automate, poorly, a great deal of customer service work; let us hope they do…
If the primary complaint is the blues that GPT-4 wrote is not that great, I think it is definitely worth the hype, given that a year before people argued that AI can never pass turing test.
Earlier quoted context omitted.
What alternative technology do you think is better? In other words, what is your frame of reference for labeling this "pretty terrible"?
Humans. After all, LLMs are designed to reason equal to or better than humans.
Earlier quoted context omitted.
GPT 4 and another LLM have given the right answer only after adding "Let's think step by step." to the original prompt. With the simpler prompt, all the answers were wrong, most of them ridiculously wrong.
I asked GPT-4 through both ChatGPT and the API and it responded correctly without any fiddling. I keep seeing comments and posts on HN that significantly downplay GPT-4's capabilities. Are people actually using GPT-4 or are they using a 3rd party service that claims to be GPT-4? I got: >Sally has 3 brothers, and each of those brothers has 2 sisters. One of those sisters is Sally herself, and the other one is Sally's…
> Sally has 2 sisters. Each of her 3 brothers has 2 sisters, and those sisters would be Sally and her 2 sisters.
> Sally (a girl) has 3 brothers. Each brother has 2 sisters. How many sisters does Sally have? The site reports every LLM as getting this wrong. But GPT4 seems to get it right for me: > Sally has 3 brothers. Since each brother has 2 sisters and Sally is one of those sisters, the other sister is the second sister for each brother. So, Sally has 1 sister.