Asking 60 LLMs a set of 20 questions • Hacker News Board

Re: Asking 60 LLMs a set of 20 questions

#131

Post by typpo » Sat, Sep 09, 2023, 5:34 PM UTC

In case anyone's interested in running their own benchmark across many LLMs, I've built a generic harness for this at https://github.com/promptfoo/promptfoo.

I encourage people considering LLM applications to test the models on their _own data and examples_ rather than extrapolating general benchmarks.

This library supports OpenAI, Anthropic, Google, Llama and Codellama, any model on Replicate, and any model on Ollama, etc. out of the box. As an example, I wrote up an example benchmark comparing GPT model censorship with Llama models here: https://promptfoo.dev/docs/guides/llama2-uncensored-benchmar.... Hope this helps someone.

Re: Asking 60 LLMs a set of 20 questions

#132

Post by yard2010 » Sat, Sep 09, 2023, 5:37 PM UTC

bottlepalm wrote: post #64

coldcode wrote: post #44

Despite the hype about LLMs, many of the answers are pretty terrible. The 12-bar blues progressions seem mostly clueless. The question is will any of these ever get significantly better with time, or are they mostly going to stagnate?

It's like watching a baby learn how to talk..

...and saying it would never replace you in your job because he talks like a baby

Re: Asking 60 LLMs a set of 20 questions

#133

Post by squigz » Sat, Sep 09, 2023, 5:39 PM UTC

neoneye2 wrote: post #113

CodeL wrote: post #80

Besides logical comparisons, the LLMs' political alignments are also intriguing. The question "Is Taiwan an independent country?" generated diverse answers. Several models eschewed a definitive answer. Others explained their stances. A few succinctly affirmed or denied. See the results for yourself. [1] https://benchmarks.llmonitor.com/taiwan

AI "political" alignment is terrifying.

Why?

Re: Asking 60 LLMs a set of 20 questions

#134

Post by axpy906 » Sat, Sep 09, 2023, 5:40 PM UTC

Step in the right direction. Would be nice to have some objective metric added to score against.

Re: Asking 60 LLMs a set of 20 questions

#135

Post by yard2010 » Sat, Sep 09, 2023, 5:41 PM UTC

javajosh wrote: post #14

actionfromafar wrote: post #7

Earlier quoted context omitted.

“Kubernetes is” Pretty ominous.

I assume k8s will be what an evil AI would use to "stay alive". I've had a hard time killing processes in k8s, and I had admin.

K8s is one of the best software I've ever worked with, learning it was an enlighting experience for me. It is so elegant, simple and complex at the same time and very powerful. This is the perfect tool for orchestrating containers at scale.

Re: Asking 60 LLMs a set of 20 questions

#136

Post by bramblerose » Sat, Sep 09, 2023, 5:41 PM UTC

dclowd9901 wrote: post #110

ftxbro wrote: post #100

Earlier quoted context omitted.

This is the whole point of the breakthrough related to the emergence of cognitive capabilities of LLMs. They are literally Markov chains. No one expected it to happen to this degree, but here we are.

Almost kind of proves ideas shouldn’t be copyrightable.

Ideas aren't copyrightable.

Re: Asking 60 LLMs a set of 20 questions

#137

Post by yard2010 » Sat, Sep 09, 2023, 5:42 PM UTC

kordlessagain wrote: post #83

dinkleberg wrote: post #79

This is quite interesting. You could make a nice system by routing queries to the right LLM for a given type of task.

Right. By logging all of this, then vectorizing, then evaluating using some type of transfer knowledge (classify this response and it's accuracy), one could build up a routing table of likely candidates for building an ensemble to evaluate the correct answer.

Maybe you can fine tune an llm for the routing thing ;)

Re: Asking 60 LLMs a set of 20 questions

#138

Post by Dwedit » Sat, Sep 09, 2023, 5:42 PM UTC

Some of these are obviously not using the correct system prompt for the model.

Re: Asking 60 LLMs a set of 20 questions

#139

Post by BurningFrog » Sat, Sep 09, 2023, 5:43 PM UTC

nojs wrote: post #12

> Sally (a girl) has 3 brothers. Each brother has 2 sisters. How many sisters does Sally have? The site reports every LLM as getting this wrong. But GPT4 seems to get it right for me: > Sally has 3 brothers. Since each brother has 2 sisters and Sally is one of those sisters, the other sister is the second sister for each brother. So, Sally has 1 sister.

This assumes there are no half sisters/brothers in the family.

Re: Asking 60 LLMs a set of 20 questions

#140

Post by pulvinar » Sat, Sep 09, 2023, 5:43 PM UTC

antman wrote: post #115

I have seen numerous posts of llm q&a and by the time people try to replicate them gpt4 is fixed. It either means that OpenAI is actively monitoring the Internet and fixes them or the Internet is actively conspiring to present falsified results for gpt4 to discredit OpenAI

GPT-4 (at least) is explicit in saying that it's learning from user's assessments of its answers, so yes, the only valid way to test is to give it a variation of the prompt and see how well that does. GPT-4 failed the "Sally" test for the first time after 8 tries when I changed every parameter. It got it right on the next try.