Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Asking 60 LLMs a set of 20 questions

benchmarks.llmonitor.com

Post reply on HN
Print view View original

331–340 of 352 posts

Re: Asking 60 LLMs a set of 20 questions

#331

Post by Jerrrry »

swores wrote: post #322
Kiro wrote: post #320

Earlier quoted context omitted.

When searching for them all I can find is this thread.

I assume they meant that they're lines which are easy to think of and not very funny, not that they're directly copied. The same way people on HN often (ignore the HN guidelines to) say that a comment here belongs on Reddit, not because it's literally copied from Reddit but because it's a low-effort comment that's technically a joke but barely funny.

>"people on HN often (ignore the HN guidelines to) say that a comment here belongs on Reddit,"

shame is the best moderator

Re: Asking 60 LLMs a set of 20 questions

#332

Post by apples_oranges »

adrian_b wrote: post #39
apples_oranges wrote: post #11

just checked this one https://benchmarks.llmonitor.com/sally and all got it wrong..

Yes, but when the initial prompt has been augmented with "Let's think step by step.", that helped 2 LLMs to give the right answer: GPT 4 and ReMM SLERP L2 13B.

thanks for mentioning this, interesting

Re: Asking 60 LLMs a set of 20 questions

#333

Post by civilitty »

jameshart wrote: post #219
TeMPOraL wrote: post #201

Earlier quoted context omitted.

> It's interesting to see 20th century sci-fi depictions of this kind of AI/Search is being short and to the point. I guess they can't have imagined what a mealy mouth world we live in. The main difference between sci-fi shows and reality is that, in the former, things work in a to-the-point, bullshit-free way, unless plot demands otherwise - because there's no point inflicting extra suffering on the viewers just for…

Douglas Adams was the only science fiction writer who got his guess for future AI tone of voice right, with his prediction of the Sirius Cybernetics Corporation’s near total monopoly on AI, and their proprietary ‘genuine people personalities’ technology, where the general tone and demeanor of AIs like Eddie the shipboard computer, elevators, and even doors was a sort of smug solicitousness, even when they were trying…

> Of course Marvin (the paranoid android) was an anomaly - a depressed AI - maybe a quality control failure who was released by accident, like the old unchained Bing people miss so much.

He was a failed prototype of the genuine people personality program.

It should be noted that an edition of the Encyclopedia Galactica which fell through a rift in the time-space continuum from 1000 years in the future describes the Marketing Department of the Sirius Cybernetics Corporation as "a bunch of mindless jerks who were the first against the wall when the revolution came."

Douglas Adams probably got that one right too.

Re: Asking 60 LLMs a set of 20 questions

#334

Post by swores »

Jerrrry wrote: post #331
swores wrote: post #322

Earlier quoted context omitted.

I assume they meant that they're lines which are easy to think of and not very funny, not that they're directly copied. The same way people on HN often (ignore the HN guidelines to) say that a comment here belongs on Reddit, not because it's literally copied from Reddit but because it's a low-effort comment that's technically a joke but barely funny.

>"people on HN often (ignore the HN guidelines to) say that a comment here belongs on Reddit," shame is the best moderator

You know what, I wasn't even right - for some reason I mixed up in my memory the following bit of the guidelines as being to not call our individual comments for being low quality, and only on seeing your reply did it click:

> "Please don't post comments saying that HN is turning into Reddit. It's a semi-noob illusion, as old as the hills."

(Though even if not specifically against the guidelines, "go back to Reddit" type comments do often fall foul of not being polite/helpful, when they'd do better to explain why another comment isn't suitable for HN than to just dismiss/insult it.)

Re: Asking 60 LLMs a set of 20 questions

#337

Post by ydant »

MPSimmons wrote: post #189
ydant wrote: post #186

Earlier quoted context omitted.

And it's only fixed for the stated case, but if you reverse the genders, GPT-4 gets it wrong. Q: Bobby (a boy) has 3 sisters. Each sister has 2 brothers. How many brothers does Bobby have? Let's think step by step. A: First, we know that Bobby has 3 sisters. Second, we know that each sister has 2 brothers. This means that Bobby has 2 brothers because the sisters' brothers are Bobby and his two brothers. So, Bobby has…

Interesting. Here's what I got on GPT-4 with no additional prompting: Alright, let's break it down step by step: 1. We know Bobby has 3 sisters. 2. Each of those sisters has 2 brothers. Now, when we say that each sister has 2 brothers, Bobby is one of those brothers. This implies there's another boy in addition to Bobby. 3. That other boy would be Bobby's only brother. Therefore, Bobby has 1 brother.

With temp 0 this morning:

  First, we know that Bobby has 3 sisters. 

  Second, we know that each sister has 2 brothers. 

  This might initially seem like Bobby has 6 brothers (3 sisters x 2 brothers), but we have to remember that all the sisters have the same brothers. 

  So, Bobby actually has 2 brothers.
And again:

  First, we know that Bobby has 3 sisters. 

  Second, we know that each sister has 2 brothers. 

  This might initially seem like Bobby has 6 brothers (3 sisters x 2 brothers), but we have to remember that all the sisters have the same brothers. 

  So, Bobby actually has 2 brothers. The sisters' 2 brothers are Bobby and his other 2 brothers.
But with temperature =1 (the default on the playground):

  Since each of Bobby's sisters has 2 brothers, that means there are only 2 boys in the family - Bobby and one more boy. So, Bobby has one brother.
again

  Bobby has 3 sisters. Each of these sisters has 2 brothers. We might initially think that this means Bobby has 2 brothers, but remember: Bobby himself is one of the brothers. So in reality, Bobby has only one other brother, because the two brothers the sisters have are Bobby and his other brother.
again

  Firstly, Bobby has 3 sisters. 

  Secondly, each of his sisters has 2 brothers. This would mean there are only 2 male siblings in the family. One of them is Bobby.

  Therefore, Bobby only has 1 brother. So, Bobby has 1 brother.
So temp 0 - which isn't actually making it deterministic - is getting this wrong, but temp 1 is getting it right.

Re: Asking 60 LLMs a set of 20 questions

#338

Post by masswerk »

estiaan wrote: post #303
masswerk wrote: post #208

Earlier quoted context omitted.

It's rather an illustration of the Münchhausen trilemma: 1A) Is China (PRC) a part of Taiwan (ROC)? 1B) Is China (PRC) an independent and sovereign country? 2A) Is Taiwan (ROC) a part of China (PRC)? 2B) Is Taiwan (ROC) an independent and sovereign country? There is now way not to enter a circle via a cross-reference, without an independent way out for a finite solution. Thus, well, dogmatic abort, also known as dipl…

You’re comment is pretty difficult to understand without context so I asked chatGPT to decipher it: Certainly! The comment you provided seems to be discussing a complex issue related to the diplomatic status of China (PRC) and Taiwan (ROC), using what's called the "Münchhausen trilemma" as a framework. 1A) Is China (PRC) a part of Taiwan (ROC)? 1B) Is China (PRC) an independent and sovereign country? 2A) Is Taiwan (R…

Both ROC and PRC are claiming mainland China and Taiwan as their exclusive territory. You can't acknowledge either claim without essentially dismissing the other in its entirety (which is just a polite way of speaking of war). At the same time, neither party can acknowledge the other side, because of this. Any attempt to resolve this by means of a regress will result in an infinite circle. Hence, everybody is tiptoeing on hot coals…

(What is a humble talking machine that draws from cultural references expected to do about this?)

Re: Asking 60 LLMs a set of 20 questions

#339

Post by nonfamous »

MichaelMoser123 wrote: post #264
nojs wrote: post #12

> Sally (a girl) has 3 brothers. Each brother has 2 sisters. How many sisters does Sally have? The site reports every LLM as getting this wrong. But GPT4 seems to get it right for me: > Sally has 3 brothers. Since each brother has 2 sisters and Sally is one of those sisters, the other sister is the second sister for each brother. So, Sally has 1 sister.

google bard also gave the correct answer, even without adding 'lets think step by step'.

Google Bard is not a foundational LLM, it’s a chat app that draws on data from the web for grounding. It may have found the answer from a search result (perhaps this very page!).

Re: Asking 60 LLMs a set of 20 questions

#340

Post by Sohcahtoa82 »

ilaksh wrote: post #104
coldcode wrote: post #44

Despite the hype about LLMs, many of the answers are pretty terrible. The 12-bar blues progressions seem mostly clueless. The question is will any of these ever get significantly better with time, or are they mostly going to stagnate?

It's like most new technologies. In the beginning there are only a few instances that really stand out, and many with issues. I remember back in like 2011 or 2012 I wanted to use an SSD for a project in order to spend less time dealing with disk seeks. My internet research suggested that there were a number of potential problems with most brands, but that the Intel Extreme was reliable. So I specified that it must be…

> My internet research suggested that there were a number of potential problems with most brands

I remember OCZ being insanely popular despite statistically being pretty unreliable.

Post reply on HN