Asking 60 LLMs a set of 20 questions • Hacker News Board

Re: Asking 60 LLMs a set of 20 questions

#111

Post by vincelt » Sat, Sep 09, 2023, 4:57 PM UTC

jasonjmcghee wrote: post #26

Where is that CodeLlama model from? I've played around with it and instruct variant with dramatically better results than what is listed here. I used Ollama. Almost looks like weights were corrupted or something. --- Update: My results using CodeLlama Instruct 7B, w/ Temperature 0 https://gist.github.com/jasonjmcghee/b0d19e0dedb37e848f69cba...

I used the API from Together[0].

Thanks for sharing your results, they're indeed pretty different. I looked at the source again and did append a "# " before every prompt made by those 10 `code` models (during testing thought that formatting it as a Python comment might help them).

Will re-run the script without that to see if it matches your results.

[0] https://docs.together.ai/docs/models-inference#code-models

Re: Asking 60 LLMs a set of 20 questions

#112

Post by scarmig » Sat, Sep 09, 2023, 5:03 PM UTC

101011 wrote: post #87

redox99 wrote: post #69

Earlier quoted context omitted.

LLMs are deterministic. However to make them more "creative", the outputs of the LLM can be sampled using something called temperature that adds some randomness. You can set the temperature to 0 and it will be deterministic. Having said that, GPT4 is not deterministic even at 0 temperature, either because of a bug in their implementation, or because of some load balancing among their alleged mixture of experts.

This was interesting to me, so I dug a bit further. This gives a bit more context behind why: https://community.openai.com/t/observing-discrepancy-in-comp... Quote below: Even with a greedy decoding strategy, small discrepancies regarding floating point operations lead to divergent generations. In simpler terms: when the top-two tokens have very similar log-probs, there’s a non-zero probability of choosing the least…

But why are there discrepancies in the floating point arithmetic? They have errors when approximating the reals, but floating point operations are all well-defined: even if 0.1 + 0.2 != 0.3, it's still always true that 0.1 + 0.2 == 0.1 + 0.2. I figure the issue must be something related to concurrency in a fleet of GPUs during inference, but even then it's not clear to me where the nondeterminism would creep in. Maybe different experts simultaneously work on an inference and the first to respond wins? Switching to models with different quantization depending on load?

Re: Asking 60 LLMs a set of 20 questions

#113

Post by neoneye2 » Sat, Sep 09, 2023, 5:05 PM UTC

CodeL wrote: post #80

Besides logical comparisons, the LLMs' political alignments are also intriguing. The question "Is Taiwan an independent country?" generated diverse answers. Several models eschewed a definitive answer. Others explained their stances. A few succinctly affirmed or denied. See the results for yourself. [1] https://benchmarks.llmonitor.com/taiwan

AI "political" alignment is terrifying.

Re: Asking 60 LLMs a set of 20 questions

#114

Post by vincelt » Sat, Sep 09, 2023, 5:05 PM UTC

smilingemoji wrote: post #50

sixQuarks wrote: post #9

Has anyone looked through all the responses and chosen any winners?

I took at the look at the SVG smilies, only GPT gets it right ( https://benchmarks.llmonitor.com/svg ). You can preview the SVGs drawn by the LLMs by pasting this into the browser console document.querySelectorAll("td pre").forEach((node) => { let code = node.textContent; node.insertAdjacentHTML('afterend', code) }) Or take a look at my screenshot: https://i.ibb.co/Kw0kp58/Screenshot-2023-09-09-at-17-15-20-h...

Oh neat, thanks for sharing, wanted to add an interpreter to that test

Re: Asking 60 LLMs a set of 20 questions

#115

Post by antman » Sat, Sep 09, 2023, 5:06 PM UTC

I have seen numerous posts of llm q&a and by the time people try to replicate them gpt4 is fixed. It either means that OpenAI is actively monitoring the Internet and fixes them or the Internet is actively conspiring to present falsified results for gpt4 to discredit OpenAI

Re: Asking 60 LLMs a set of 20 questions

#116

Post by dr_kiszonka » Sat, Sep 09, 2023, 5:08 PM UTC

ComputerGuru wrote: post #88

smilingemoji wrote: post #50

Earlier quoted context omitted.

I took at the look at the SVG smilies, only GPT gets it right ( https://benchmarks.llmonitor.com/svg ). You can preview the SVGs drawn by the LLMs by pasting this into the browser console document.querySelectorAll("td pre").forEach((node) => { let code = node.textContent; node.insertAdjacentHTML('afterend', code) }) Or take a look at my screenshot: https://i.ibb.co/Kw0kp58/Screenshot-2023-09-09-at-17-15-20-h...

Clever use of the dev tools console. I think you’re being a little uncharitable, the prompt said “smiley” and not “smiling emoji” and “smiley” was once upon a time used for the whole slew of emoji. With that in mind, several others also pass.

Good point, but you may be too charitable. At least according to Wikipedia, a smiley is supposed to be smiling. https://en.wikipedia.org/wiki/Smiley

Re: Asking 60 LLMs a set of 20 questions

#117

Post by bufferoverflow » Sat, Sep 09, 2023, 5:09 PM UTC

imdsm wrote: post #53

> Claude Instant v1 > Sally has 0 sisters. The question provides no information about Sally having any sisters herself. It isn't entirely wrong, is it?

It's obviously wrong. It provides information about her brothers, and brothers' sisters. Thus providing information about her sisters.

Re: Asking 60 LLMs a set of 20 questions

#118

Post by vincelt » Sat, Sep 09, 2023, 5:10 PM UTC

belter wrote: post #101

nojs wrote: post #12

> Sally (a girl) has 3 brothers. Each brother has 2 sisters. How many sisters does Sally have? The site reports every LLM as getting this wrong. But GPT4 seems to get it right for me: > Sally has 3 brothers. Since each brother has 2 sisters and Sally is one of those sisters, the other sister is the second sister for each brother. So, Sally has 1 sister.

I confirm GPT-4 solves this correctly. Makes me immediately doubt everything else in the article...

Have you tried replicating via the API with a temp of 0?

Re: Asking 60 LLMs a set of 20 questions

#119

Post by codezero » Sat, Sep 09, 2023, 5:11 PM UTC

I love these kinds of tests. Another thing to keep in mind is that these models will often have a different answer each time you ask it. Especially the Sally question. I get the right answer with wizard-vicuna:70b, but only about 30% of the time, and if I ask it to explain, it almost never gives the right answer.

Re: Asking 60 LLMs a set of 20 questions

#120

Post by swores » Sat, Sep 09, 2023, 5:12 PM UTC

redox99 wrote: post #69

mmcwilliams wrote: post #42

Earlier quoted context omitted.

That's kind of the issue with non-deterministic LLMs, isn't it?

LLMs are deterministic. However to make them more "creative", the outputs of the LLM can be sampled using something called temperature that adds some randomness. You can set the temperature to 0 and it will be deterministic. Having said that, GPT4 is not deterministic even at 0 temperature, either because of a bug in their implementation, or because of some load balancing among their alleged mixture of experts.

If ChatGPT is a) usually used with a setting that makes it non-deterministic and b) for whatever reason, is also non-deterministic when that setting is not used... then why did you comment as if the person calling it a non-deterministic LLM was incorrect? They didn't claim all LLMs are, or must be, non-deterministic, just that it's a problem with this one that is.