Asking 60 LLMs a set of 20 questions • Hacker News Board

Re: Asking 60 LLMs a set of 20 questions

#91

Post by layer8 » Sat, Sep 09, 2023, 4:11 PM UTC

imdsm wrote: post #53

> Claude Instant v1 > Sally has 0 sisters. The question provides no information about Sally having any sisters herself. It isn't entirely wrong, is it?

It is wrong, at the very least, in the sense that it claims no information was provided relating to sisters, while still inferring a definite number of sisters. If no information is provided, it should instead say that the number of sisters is unknown.

Re: Asking 60 LLMs a set of 20 questions

#92

Post by trebligdivad » Sat, Sep 09, 2023, 4:12 PM UTC

Odd that a bunch share the same vacation jokes or varieties of them; is there some small corpus of jokes?

Re: Asking 60 LLMs a set of 20 questions

#93

Post by phillipcarter » Sat, Sep 09, 2023, 4:21 PM UTC

tgv wrote: post #72

phillipcarter wrote: post #19

Earlier quoted context omitted.

Nondeterminism strikes again! But yes, I would expect GPT-4 to get this right most of the time.

Saying "Sorry, I was non-deterministic" to your teacher won't do much for your grade.

Sure, but what is your point? This is about evaluating LLM outputs, not grade school.

Re: Asking 60 LLMs a set of 20 questions

#94

Post by ailef » Sat, Sep 09, 2023, 4:21 PM UTC

Gunnerhead wrote: post #16

I get frustrated when I tell an LLM “reply only with x” and then rather than responding “x”, it still responds with “Sure thing! Here’s x” or some other extra words.

Have you tried setting a very low temperature, possibly zero? I've had good success with it.

Re: Asking 60 LLMs a set of 20 questions

#95

Post by rootusrootus » Sat, Sep 09, 2023, 4:25 PM UTC

rubymamis wrote: post #22

sixQuarks wrote: post #9

Has anyone looked through all the responses and chosen any winners?

GPT4 seems to me to be the best. Undi95/ReMM-SLERP-L2-13B the runner up.

Clearly the prompt construction and initial instructions are critically important here. Without that, the ReMM-SLERP-L2-13B model produces awful results. Blank answers about half the time. For lay users, ChatGPT remains the undisputed winner. Although I do see reasonably good results with the more recent llama-2 70B variations, which are plausibly useful a majority of the time.

Re: Asking 60 LLMs a set of 20 questions

#96

Post by DylanDmitri » Sat, Sep 09, 2023, 4:26 PM UTC

maweaver wrote: post #58

salil999 wrote: post #49

Earlier quoted context omitted.

Humans. After all, LLMs are designed to reason equal to or better than humans.

By "Humans", I assume you mean something like "adult humans, well-educated in the relevant fields". Otherwise, most of these responses look like they would easily beat most humans.

I think most high-school educated adults, with the ability to make a couple web searches, would do fine on all these questions. It would take the humans minutes instead of seconds because they don't have the internet memorized.

Me, Kubernetes Haikus, time taken 84 seconds:

----------

Kubernetes rules

With its smooth orchestration

You can reach web scale

----------

Kubernetes sucks

Lost in endless YAML hell

Why is it broken?

Re: Asking 60 LLMs a set of 20 questions

#97

Post by jonwinstanley » Sat, Sep 09, 2023, 4:32 PM UTC

nojs wrote: post #12

> Sally (a girl) has 3 brothers. Each brother has 2 sisters. How many sisters does Sally have? The site reports every LLM as getting this wrong. But GPT4 seems to get it right for me: > Sally has 3 brothers. Since each brother has 2 sisters and Sally is one of those sisters, the other sister is the second sister for each brother. So, Sally has 1 sister.

I wouldn’t expect an LLM to get this right unless it had been trained on a solution.

Am I wrong to think that? Are LLMs in the future going to be able to “think through” actual logic problems?

Re: Asking 60 LLMs a set of 20 questions

#98

Post by IshKebab » Sat, Sep 09, 2023, 4:33 PM UTC

NoraCodes wrote: post #52

smokel wrote: post #45

Earlier quoted context omitted.

What alternative technology do you think is better? In other words, what is your frame of reference for labeling this "pretty terrible"?

Given that people are already firing real human workers to replace them with worse but cheaper LLMs, I'd argue that we're not talking about a competing technology, but that the competition is simply not firing your workforce. And, as an obligate customer of many large companies, you should be in favor of that as well. Most companies already automate, poorly, a great deal of customer service work; let us hope they do…

But they're only firing humans in specific cases where LLMs can compete with them. LLMs aren't equally good at all tasks.

Re: Asking 60 LLMs a set of 20 questions

#99

Post by PrayagBhakar » Sat, Sep 09, 2023, 4:34 PM UTC

jonwinstanley wrote: post #97

nojs wrote: post #12

> Sally (a girl) has 3 brothers. Each brother has 2 sisters. How many sisters does Sally have? The site reports every LLM as getting this wrong. But GPT4 seems to get it right for me: > Sally has 3 brothers. Since each brother has 2 sisters and Sally is one of those sisters, the other sister is the second sister for each brother. So, Sally has 1 sister.

I wouldn’t expect an LLM to get this right unless it had been trained on a solution. Am I wrong to think that? Are LLMs in the future going to be able to “think through” actual logic problems?

You could achieve thinking though logical problems by adding chain of thought or tree of thought. Then the LLM will try to break stuff down into logic it’s already seen.

Re: Asking 60 LLMs a set of 20 questions

#100

Post by ftxbro » Sat, Sep 09, 2023, 4:41 PM UTC

jonwinstanley wrote: post #97

nojs wrote: post #12

> Sally (a girl) has 3 brothers. Each brother has 2 sisters. How many sisters does Sally have? The site reports every LLM as getting this wrong. But GPT4 seems to get it right for me: > Sally has 3 brothers. Since each brother has 2 sisters and Sally is one of those sisters, the other sister is the second sister for each brother. So, Sally has 1 sister.

I wouldn’t expect an LLM to get this right unless it had been trained on a solution. Am I wrong to think that? Are LLMs in the future going to be able to “think through” actual logic problems?

This is the whole point of the breakthrough related to the emergence of cognitive capabilities of LLMs. They are literally Markov chains. No one expected it to happen to this degree, but here we are.