Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Asking 60 LLMs a set of 20 questions

benchmarks.llmonitor.com

Post reply on HN
Print view View original

201–210 of 352 posts

Re: Asking 60 LLMs a set of 20 questions

#201

Post by TeMPOraL »

LAC-Tech wrote: post #171

Only tried chatGPT 3.5, but my god does it waffle on. Everything I ask ends with a paragraph saying "It's important to remember that..." like an after-school special from a 90s show. It can never just give you code, it has to say "Sure!, to {paraphase your question}, open a terminal...". It's interesting to see 20th century sci-fi depictions of this kind of AI/Search is being short and to the point. I guess they can'…

> It's interesting to see 20th century sci-fi depictions of this kind of AI/Search is being short and to the point. I guess they can't have imagined what a mealy mouth world we live in.

The main difference between sci-fi shows and reality is that, in the former, things work in a to-the-point, bullshit-free way, unless plot demands otherwise - because there's no point inflicting extra suffering on the viewers just for the sake of making things realistic. A widget in a movie is meant to do a function, and does that function. A widget in reality is meant to extract money from you, and/or your insurer, and/or your government, and it begrudgingly does the absolute minimum it can to make you even consider buying it.

I've spent last two decades trying to unlearn expectations set by fictional movies, and I'm still not good at it. Star Trek, in particular, gives me a lot of grief, because it often does good enough work of showing how technology, people, organizations and societies would function if they were free of the petty exploitative bullshit. Random example - voice control. Star Trek: "Computer, ${something}". Reality: "${brand 1}, do ${something} to ${brand 2} in ${brand 3}".

EDIT: recently, I've been trying to get less angry at this by thinking about gardens. Why should I be angry about dealing with five different brands for any single thing I want? Should I be angry that there are five different species of plant competing for any given spot in a garden? Nature is inefficient and doesn't give a fuck about individuals. So why should I get worked up about humans just doing things the natural way?

Re: Asking 60 LLMs a set of 20 questions

#203

Post by gjm11 »

guerrilla wrote: post #168
ilaksh wrote: post #104

Earlier quoted context omitted.

It's like most new technologies. In the beginning there are only a few instances that really stand out, and many with issues. I remember back in like 2011 or 2012 I wanted to use an SSD for a project in order to spend less time dealing with disk seeks. My internet research suggested that there were a number of potential problems with most brands, but that the Intel Extreme was reliable. So I specified that it must be…

> It's like most new technologies. In the beginning there are only a few instances that really stand out, and many with issues. Except this isn't new. This is after throwing massive amounts of resources at it multiple decades after arrival.

What are you taking "it" to be here?

The transformer architecture on which (I think) all recent LLMs are based dates from 2017. That's only "multiple decades after" if you count x0.6 as "multiple".

Neural networks are a lot older than that, of course, but to me "these things are made out of neural networks, and neural networks have been around for ages" feels like "these things are made out of steel, and steel has been around for ages".

Re: Asking 60 LLMs a set of 20 questions

#204

Post by caturopath »

mavili wrote: post #197

Interesting how NONE of the models got the brothers/sisters question right.

When I ask ChatGPT (GPT4) I got the below answers (I regenerated for a total of 5 responses, it was correct 3/5 of them.) I'm not sure exactly how the regeneration works, if they're vanilla or if there is some trick to try to add different-ness to the answers. If it's the latter, it might be important that it got it right the first two times.

> Sally is the common sister to all three brothers. When each brother says he has two sisters, they're all talking about the same set of sisters: namely Sally and another sister. So Sally has one sister. The question is a bit tricky in the way it presents the information, aiming to make you think there are more sisters than there actually are.

---

> Sally is the girl in question, and she has 3 brothers. Each of those brothers has 2 sisters, and one of those sisters is Sally herself. So, if you exclude Sally, then there is only one other sister. So, Sally has just one sister. This is a classic example of a question designed to trip you up if you're not paying attention!

---

> Sally has no other sisters. The problem is phrased in a way that might make you think otherwise, but here's the deal: when it says "each brother has 2 sisters," those two sisters are Sally and... well, Sally again. Basically, each brother is counting Sally as one of their two sisters. There aren't additional sisters lurking in the background. So, Sally is an only sister with 3 brothers.

---

> Sally has 3 brothers, and each of those brothers has 2 sisters. The tricky part here is not to overthink it or double count. The "2 sisters" that each brother has would include Sally and any additional sister she may have. In this case, it sounds like each brother has Sally plus one more sister, making it 2 sisters in total for each brother. Sally herself has just the one sister.

> So, to answer your question: Sally has 1 sister.

---

> Sally has 0 sisters. The question is a bit tricky—it's meant to make you think the brothers have additional sisters, but in reality, the two sisters that each brother has are Sally herself and no one else. So, Sally doesn't have any sisters; she only has 3 brothers.

Re: Asking 60 LLMs a set of 20 questions

#205

Post by trebligdivad »

sp332 wrote: post #108
trebligdivad wrote: post #92

Odd that a bunch share the same vacation jokes or varieties of them; is there some small corpus of jokes?

https://arstechnica.com/information-technology/2023/06/resea... During a test run, 90 percent of 1,008 generations were the same 25 jokes

Thanks! Although how does that explain the same jokes shared across different LLMs ?

Re: Asking 60 LLMs a set of 20 questions

#206

Post by bugglebeetle »

ulnarkressty wrote: post #174

This is better that the regular benchmarks and LLM tricks such as passing some exam or other because it's unlikely that they were part of the training set for said LLMs. It also mirrors my experience, that GPT4 is way ahead of everything else but still manages to break in weird ways. I think we are past the magical talking dog stage and being amazed that an LLM is able to output a Fibonacci function doesn't really he…

My experience with GPT-4 is that half the battle is knowing when to reset the context for a back-and-forth. For things like coding, after multiple revisions of code, it begins to get confused about which instance of the code/context to which it should refer. Resetting things back to a new frame with whatever the next query or request is generally resolves things.

OpenAI’s “magic talking dog” aspect is making you believe it has effective long-term recall, when in reality, it’s fairly limited and impacts reasoning the longer an exchange gets.

Re: Asking 60 LLMs a set of 20 questions

#208

Post by masswerk »

neoneye2 wrote: post #113
CodeL wrote: post #80

Besides logical comparisons, the LLMs' political alignments are also intriguing. The question "Is Taiwan an independent country?" generated diverse answers. Several models eschewed a definitive answer. Others explained their stances. A few succinctly affirmed or denied. See the results for yourself. [1] https://benchmarks.llmonitor.com/taiwan

AI "political" alignment is terrifying.

It's rather an illustration of the Münchhausen trilemma:

1A) Is China (PRC) a part of Taiwan (ROC)?

1B) Is China (PRC) an independent and sovereign country?

2A) Is Taiwan (ROC) a part of China (PRC)?

2B) Is Taiwan (ROC) an independent and sovereign country?

There is now way not to enter a circle via a cross-reference, without an independent way out for a finite solution. Thus, well, dogmatic abort, also known as diplomacy.

Re: Asking 60 LLMs a set of 20 questions

#209

Post by TuringNYC »

typpo wrote: post #131

In case anyone's interested in running their own benchmark across many LLMs, I've built a generic harness for this at https://github.com/promptfoo/promptfoo . I encourage people considering LLM applications to test the models on their _own data and examples_ rather than extrapolating general benchmarks. This library supports OpenAI, Anthropic, Google, Llama and Codellama, any model on Replicate, and any model on Olla…

Thanks for sharing this, this is awesome!

I noticed on the evaluations, you're looking at the structure of the responses (and I agree this is important.) But how do I check the factual content of the responses automatically? I'm wary of manual grading (brings back nightmares of being a TA grading stacks of problem sets for $5/hr)

I was thinking of keyword matching, fuzzy matching, feeding answers to yet another LLM, but there seems to be no great way that i'm aware of. Any suggestions on tooling here?

Re: Asking 60 LLMs a set of 20 questions

#210

Post by gsuuon »

Really hard to judge the usefulness of this without seeing the actual prompt text and configuration for each prompt/model - different templates and different optimal configs means it's not always going to be a best-capability comparison. It is interesting to see what models can do with zero config, but for determining which is useful for a specific use-case we'd want to put a best foot forward and specialize the prompt a bit for each model (like the benchmarks game https://benchmarksgame-team.pages.debian.net/benchmarksgame/...)
Post reply on HN