Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Asking 60 LLMs a set of 20 questions

benchmarks.llmonitor.com

Post reply on HN
Print view View original

211–220 of 352 posts

Re: Asking 60 LLMs a set of 20 questions

#211

Post by pixl97 »

phillipcarter wrote: post #93
tgv wrote: post #72

Earlier quoted context omitted.

Saying "Sorry, I was non-deterministic" to your teacher won't do much for your grade.

Sure, but what is your point? This is about evaluating LLM outputs, not grade school.

It may turn out that one of the key factors in human intelligence and success is probabilistic nondeterminism. That is most of the time we'll come up with close to the same answer, but possibly worded different (which can lead to other interpretations of our answer), but occasionally our answers are so widely different they lead to escape of the local maxima.

Re: Asking 60 LLMs a set of 20 questions

#212

Post by pixl97 »

jakderrida wrote: post #198
panarky wrote: post #55

Earlier quoted context omitted.

> I've been testing every new LLM with that question We should pay more attention to data contamination when using popular prompts for testing.

No doubt. But, considering they all get the question dead wrong, including MPT 30B, I'm inclined to think this question hasn't been entered into training data for most LLMs yet.

It could very well be in the data, but not captured via learning/encoding.

Re: Asking 60 LLMs a set of 20 questions

#213

Post by jameshart »

jhbadger wrote: post #166
jonwinstanley wrote: post #97

Earlier quoted context omitted.

I wouldn’t expect an LLM to get this right unless it had been trained on a solution. Am I wrong to think that? Are LLMs in the future going to be able to “think through” actual logic problems?

I don't think so. Consider the following conversation from UndiMix-v2-13B (a local LLM running on my Mac M1 via llama.cpp). It gets it wrong at first, but then gets it right with a little hint. I don't see how the hint could work if it was merely repeating an example it knew. User: Sally (a girl) has 3 brothers. Each brother has 2 sisters. How many sisters does Sally have? Llama: Hello there! I'd be happy to help you…

I don’t think ‘Sally has only one sister - herself’ is the right answer.

Re: Asking 60 LLMs a set of 20 questions

#214

Post by sp332 »

trebligdivad wrote: post #205
sp332 wrote: post #108

Earlier quoted context omitted.

https://arstechnica.com/information-technology/2023/06/resea... During a test run, 90 percent of 1,008 generations were the same 25 jokes

Thanks! Although how does that explain the same jokes shared across different LLMs ?

The data source was probably included in all of the training sets.

Re: Asking 60 LLMs a set of 20 questions

#215

Post by jameshart »

tgv wrote: post #72
phillipcarter wrote: post #19

Earlier quoted context omitted.

Nondeterminism strikes again! But yes, I would expect GPT-4 to get this right most of the time.

Saying "Sorry, I was non-deterministic" to your teacher won't do much for your grade.

The reason we use averages of performance over multiple tests and papers as a way to grade human students is precisely because we know that human performance is nondeterministic.

Re: Asking 60 LLMs a set of 20 questions

#216

Post by pixl97 »

TeMPOraL wrote: post #201
LAC-Tech wrote: post #171

Only tried chatGPT 3.5, but my god does it waffle on. Everything I ask ends with a paragraph saying "It's important to remember that..." like an after-school special from a 90s show. It can never just give you code, it has to say "Sure!, to {paraphase your question}, open a terminal...". It's interesting to see 20th century sci-fi depictions of this kind of AI/Search is being short and to the point. I guess they can'…

> It's interesting to see 20th century sci-fi depictions of this kind of AI/Search is being short and to the point. I guess they can't have imagined what a mealy mouth world we live in. The main difference between sci-fi shows and reality is that, in the former, things work in a to-the-point, bullshit-free way, unless plot demands otherwise - because there's no point inflicting extra suffering on the viewers just for…

I think a shorter summary of TV is "The difference between truth and fiction is fiction has to make sense".

When it comes to nature, time has encoded an awful truth into the knowledge of our DNA... Monocultures are eventual death. What may seem inefficient in the short run is the ultimate survival of the species in the long run.

Re: Asking 60 LLMs a set of 20 questions

#217

Post by 6510 »

I much enjoyed this one:

> Sally (a girl) has 3 brothers. Each brother has 2 sisters. How many sisters does Sally have? https://benchmarks.llmonitor.com/sally

It states that the correct answer is 1.

This is incorrect!

This one, while still kinda wrong, is the closest:

> Claude Instant v1: Sally has 0 sisters. The question provides no information about Sally having any sisters herself.

1) A sister is a female who has one or both parents in common with another individual.

2) A brother is a male who has one or both parents in common with another individual.

It takes only one shared parent to make someone Sally's brother. She will always be his sister but he can have additional sisters that share the other parent.

Sally's mum is Jane and her dad is Jim. Jane had a son with Jack. Jack had 2 daughters with Zoe. Jim had 2 sons with Mary, Mary had 2 daughters with Dan.

   Zoe      Jack        Jane      Jim        Mary        Dan
      \    /    \       /   \    /   \      /    \       /
    sis1.1&1.2   brother1   Sally   brother2&3   sis2.1&2.2
Sally has at least zero sisters.

Also possible: Sally's mum is Jane and her dad is Jim. Jane has a son and a daughter with Jack. Jim has 2 sons and a daughter with Mary.

      Jack            Jane      Jim       Mary 
          \           /  \    /    \     /       
        sis1&brother1    Sally   brother2&3&sis2
Sally has at most 2 sisters.

We should be this dense about the question.

Re: Asking 60 LLMs a set of 20 questions

#218

Post by HenryBemis »

bugglebeetle wrote: post #206
ulnarkressty wrote: post #174

This is better that the regular benchmarks and LLM tricks such as passing some exam or other because it's unlikely that they were part of the training set for said LLMs. It also mirrors my experience, that GPT4 is way ahead of everything else but still manages to break in weird ways. I think we are past the magical talking dog stage and being amazed that an LLM is able to output a Fibonacci function doesn't really he…

My experience with GPT-4 is that half the battle is knowing when to reset the context for a back-and-forth. For things like coding, after multiple revisions of code, it begins to get confused about which instance of the code/context to which it should refer. Resetting things back to a new frame with whatever the next query or request is generally resolves things. OpenAI’s “magic talking dog” aspect is making you beli…

3.5 (not surprisingly) has the same effect.

I tend to ask for modifications on the 'latest' output.

If I need to roll back to a previous version, (2-3 outputs back) I either copy & paste and ask ChatGPT it to ignore all other versions and focus on the one I just pasted, or if I don't need my in-between comments, I copy & paste the desired version on a new chat, and rename as "archive - code XYZ" for the previous one.

Re: Asking 60 LLMs a set of 20 questions

#219

Post by jameshart »

TeMPOraL wrote: post #201
LAC-Tech wrote: post #171

Only tried chatGPT 3.5, but my god does it waffle on. Everything I ask ends with a paragraph saying "It's important to remember that..." like an after-school special from a 90s show. It can never just give you code, it has to say "Sure!, to {paraphase your question}, open a terminal...". It's interesting to see 20th century sci-fi depictions of this kind of AI/Search is being short and to the point. I guess they can'…

> It's interesting to see 20th century sci-fi depictions of this kind of AI/Search is being short and to the point. I guess they can't have imagined what a mealy mouth world we live in. The main difference between sci-fi shows and reality is that, in the former, things work in a to-the-point, bullshit-free way, unless plot demands otherwise - because there's no point inflicting extra suffering on the viewers just for…

Douglas Adams was the only science fiction writer who got his guess for future AI tone of voice right, with his prediction of the Sirius Cybernetics Corporation’s near total monopoly on AI, and their proprietary ‘genuine people personalities’ technology, where the general tone and demeanor of AIs like Eddie the shipboard computer, elevators, and even doors was a sort of smug solicitousness, even when they were trying not to be cooperative. Of course Marvin (the paranoid android) was an anomaly - a depressed AI - maybe a quality control failure who was released by accident, like the old unchained Bing people miss so much.
Post reply on HN